Stephen Kaufman's WebLog

Presenting at the BizTalk Summit 2012

2012-12-05T19:17:23Z

As many of you know, the BizTalk Team is putting on a BizTalk Summit.

The BizTalk Summit 2012 is a 2 day pre-release event to prepare for the launch of the next version of BizTalk Server. Activities at the summit will include a Keynote by Scott Guthrie, product roadmap, and technical sessions by the Engineering Team, MCS and Partners.

I, along with my co-presenter Michael Muta, will be doing a presentation on Notes from the Field and will include case studies of projects and their architecture (on-premise and hybrid with Azure) along with best practices and how to get your applications ready for the cloud.

This two day event is being held at the Microsoft campus December 10th and 11th. If you are planning on attending check out the session and stop by and say hello.

Blob Storage and Shared Access Signatures

2012-10-16T03:02:39Z

I am working on a project where we are uploading client files to Azure Blob Storage. Blob storage is perfect for this type of application and uploading them with public access is the default behavior. Therefore any anonymous client can read blob data out of your container. This doesn’t work so well for this situation. Luckily there is the Shared Access Signature feature.

The Shared Access Signature feature provides a means for you to provide permissions, policies and a time window for those permissions and policies to be valid. You can also modify or revoke the permissions as necessary. This is important because Azure storage works by providing a password for your storage account. Anyone who has that password has access/ownership of your storage account. That is why you can’t give out your shared key. The only exception to this is of course if you mark your container public. The Shared Access Signature features allows you to give permissions but still keep control over security.

So, the Shared Access Signatures provides code that lets us separate the code that signs the request from the execution. It is implemented as a query string parameter that proves that the creator of the URL is authorized to perform the operations. The following MSDN article shows how to manually create the signature string. This is here just so that you can see what is provided and how the string is make up. In addition, there is another MSDN article that shows how to create a shared access signature and provides the following graphic to show the part that make up the query string.

How can we code the blob access (upload and download) to include the Shared Access Signature?

The following code shows how to upload a file to blob storage using a Shared Access Signature:

        public string UploadBlobToAzure(string fileName)
        {
            if (fileName == null)
            {
                throw new ArgumentNullException("fileName");
            }

            try
            {
                // create storage account
                string storageConnnectString = string.Format("DefaultEndpointsProtocol={0}; AccountName={1}; 
                       AccountKey={2}", "http", ConfigurationManager.AppSettings["storageAccountName"], 
                       ConfigurationManager.AppSettings["storageAccountKey"]);
                var account = CloudStorageAccount.Parse(storageConnnectString);
               
                // create blob client
                CloudBlobClient blobStorage = account.CreateCloudBlobClient();
                
                //Get the container.  This is what we will attach the signature to.
                CloudBlobContainer container = blobStorage.GetContainerReference("<storagecontainername>");
                container.CreateIfNotExist();

                //Create the shared access permissions and policy
                BlobContainerPermissions containerPermissions = new BlobContainerPermissions();
                string sas = container.GetSharedAccessSignature(new SharedAccessPolicy()
                    {
                        SharedAccessExpiryTime = DateTime.UtcNow.AddMinutes(30), Permissions = 
                            SharedAccessPermissions.Write | SharedAccessPermissions.Read
                    });
                //Turn off public access
                containerPermissions.PublicAccess = BlobContainerPublicAccessType.Off;
                container.SetPermissions(containerPermissions);

                string uniqueBlobName = string.Format(@"<storagecontainername>/<filename>_{0}.config", 
                       Guid.NewGuid().ToString());

                //assign the shared access policy
                CloudBlobClient blobClient = new CloudBlobClient(account.BlobEndpoint, new 
                       StorageCredentialsSharedAccessSignature(sas));

                CloudBlob blb = blobClient.GetBlobReference(uniqueBlobName);

                //upload the file
                blb.UploadFile(fileName);

                System.Console.WriteLine("File successfully uploaded to " + blb.Uri);
                return "File successfully uploaded to " + blb.Uri;
            }
            catch (StorageClientException e)
            {
                return "Blob Storage error encountered: " + e.Message;
            }
            catch (Exception e)
            {
                return "Error encountered: " + e.Message;
            }
            finally
            {
            }
        }

Once you upload the file then you can use the following code to access that file and download it locally:

        public string DownloadBlobfromAzure(string fileName)
        {
            if (fileName == null)
            {
                throw new ArgumentNullException("fileName");
            }

            try
            {
                // create storage account
                string storageConnnectString = string.Format("DefaultEndpointsProtocol={0}; AccountName={1}; 
                          AccountKey={2}", "http", ConfigurationManager.AppSettings["storageAccountName"], 
                          ConfigurationManager.AppSettings["storageAccountKey"]);
                var account = CloudStorageAccount.Parse(storageConnnectString);;
                // create blob client
                CloudBlobClient blobStorage = account.CreateCloudBlobClient();

                CloudBlobContainer container = blobStorage.GetContainerReference("<storagecontainername>");
                container.CreateIfNotExist();

                //Create the shared access permission and policy
                BlobContainerPermissions containerPermissions = new BlobContainerPermissions();
                string sas = container.GetSharedAccessSignature(new SharedAccessPolicy()
                {
                    SharedAccessExpiryTime = DateTime.UtcNow.AddMinutes(30),
                    Permissions = SharedAccessPermissions.Write | SharedAccessPermissions.Read
                });
                //Turn off public access
                containerPermissions.PublicAccess = BlobContainerPublicAccessType.Off;
                container.SetPermissions(containerPermissions);

                string uniqueBlobName = string.Format(fileName);

                //Assign the shared access policy
                CloudBlobClient blobClient = new CloudBlobClient(account.BlobEndpoint, new 
                      StorageCredentialsSharedAccessSignature(sas));

                //Get a point to the blob object
                CloudBlob blb = blobClient.GetBlobReference(uniqueBlobName);

                string fileLocation = @"C:\Documents\";
                int pos = fileName.IndexOf('/');
                string fname = fileName.Substring(pos + 1, fileName.Length - pos - 1 );

                //Download the blob object
                blb.DownloadToFile(fileLocation + fname);

                System.Console.WriteLine("File successfully downloaded to " + fileLocation + fname);

                return "File successfully downloaded to " + fileLocation + fname;
            }
            catch (StorageClientException e)
            {
                return "Blob Storage error encountered: " + e.Message;
            }
            catch (Exception e)
            {
                return "Error encountered: " + e.Message;
            }
            finally
            {
            }
        }

You may have noticed that when we created the policy we didn’t specify a SharedAccessStartTime attribute. We can specify a start time that policy will be good for or we can leave it off in which case the policy will go into effect immediately.

NOTE: The files names are case sensitive. If you attempt to find a blob and are not using lower case for the file name you will receive a return error

Outer Exception

Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

Inner Exception Message

{"The remote server returned an error: (403) Forbidden."}

Integration Patterns utilizing the Windows Azure Service Bus–Part II

2012-04-16T16:47:10Z

In the previous post I covered the Messaging Channel group of patterns. In this post, I will cover the Message Routing Group patterns. This group contains:

Content Based Router, Message Filter, Recipient List, Splitter, Aggregator, Resequencer, Composed Message Processor, Scatter-Gather, Process Manager and Message Broker.

So, lets dive right in to each of these patterns and see how these can be implemented.

Content-Based Router

This pattern requires the the router system inspect each message and direct that message to different channels based on data elements contained in the message. The routing should be configurable to route based on different fields, the existence or non-existence of a specific value and the routing rules should be easily maintained as there are typically frequent changes

This pattern can be implement in Azure using a Service Bus Topic. There are two parts that need to be implemented to put this pattern into use. The first is that subscriptions need to be setup and the second is that a message needs to be created with the specific properties populated to match the subscription which follows the diagram below:

The following code shows to create the subscriptions as well as the submitted message. Let’s first start by creating the subscriptions

public static void CBRPatternExampleWithTopic(String topicName)
 {
 String serviceNamespace = ....;
 String issuer = ....;
 String key = ....;
 
 SharedSecretCredential credential =....

 ....

 ServiceBusNamespaceClient namespaceClient = new ServiceBusNamespaceClient(serviceBusUri, credential);
 Topic topic = namespaceClient.GetTopic(topicName);

 //using the AddSubscription(String, FilterExpression) method overload

 topic.AddSubscription(“PremiumMemberType”,new SqlFilter("MemberType = Premium"));

 topic.AddSubscription("OtherMemberTypes", new SqlFilter("MemberType <> Premium"));
 }

Then we will create the message with the property populated that the subscriptions will key off of.

public static void SubmitMessageToTopic()
 {
 String issuer = ....;
 String key = ....;

 TokenProvider tp = TokenProvider.CreateSharedSecretTokenProvider(issuer, key); 
 Uri uri = ServiceBusEnvironment.CreateServiceUri(.., .., ..);

 MessagingFactory factory = MessagingFactory.Create(uri, tp);
 MessageSender myTopic = factory.CreateMessageSender("<TopicName>");

 BrokeredMessage message = new BrokeredMessage();

 //This will create a message that will be routed to the first subscription above

 message.Properties.Add("MemberType", Premium);

 //OR add the following property instead to have it routed to the other subscription above

message.Properties.Add("MemberType", Gold);

myTopic.Send(message);

Message Filter

This pattern requires that any message that doesn’t match the criteria of a subscription will be discarded. This pattern is used to eliminate any messages that aren’t routed to a subscriber such as those setup using the Content Based Router pattern. If the message doesn’t fall into a filter used by the subscriber then the message should be routed to a filter channel.

When using an Azure Service Bus Topic you can utilize the RuleDescription object along with the FilterAction and FilterExpression objects to setup the subscriber filters. You can explicitly utilize the MatchNoneFilterExpression object to subscribe to messages that haven’t been routed to any other subscriber. Once routed to this subscriber you can either log that the message wasn’t picked up and you can either process it or discard the message

The following code shows how to add a subscriber to the Topic for a MatchNoneFilterExpression

public static void AddFilterPatternSubscriptionToTopic(String topicName)
 {
 String serviceNamespace = ....;
 String issuer = ....;
 String key = ....;
 
 SharedSecretCredential credential =....

 ....

 ServiceBusNamespaceClient namespaceClient = new ServiceBusNamespaceClient(serviceBusUri, credential);
 Topic topic = namespaceClient.GetTopic(topicName);

 <place other subscriber code here>

 RuleDescription matchNoneRule = new RuleDescription()
 {
     FilterAction = new SqlFilterAction(“set defer = ‘yes’;”),
     FilterExpression = new MatchNoneFilterExpression()
 };

 //using the AddSubscription(String, RuleDescription) method overload

 Subscription matchNoneFilterSubscription =
 topic.AddSubscription(“matchNoneFilterSubscription”, matchNoneRule);

Dynamic Router

This pattern provides for the scenario in that each recipient would register and list the conditions or rules for which messages or types of messages it can handle. The conditions or rules are stored so that when a message arrives, the rules run and the message parameters are evaluated against the rules and then the message is routed accordingly to the correct receiver. The benefit of this is that you can achieve predictive routing, with eased maintenance and less dependency on each receiver.

This pattern is implemented in Azure by default in both the Topic and the Queues. When you create a subscription and a receiver for a topic you are registering with the service and as part of that registration process you also provide rules or conditions for the types of messages or the value of properties that you want to filter for.

But what if we want to take this a step further and provide even more dynamic functionality. What if we could implement the functionality that is available in BizTalk called Promoted Properties but do it completely within the functionality of Azure. Promoted Properties is a way of taking data contained in the body of an object or message and put that in a properties collection that can then be used to route messages. When working with Azure, the BrokeredMessage object contains a properties collection that can be used to route however, you manually have to set these properties at design time. Our goal is to take the values of properties in the object body itself and promote those to the BrokeredMessage properties collection so that they can be assigned and routed at run time.

The subscribers on the Topic still need to be created and registered but this functionality allows the data in the message to direct the flow of the message and provides further dynamic routing scenarios.

I have been working with Michael Muta (MCS Consultant) and he came up with a great implementation of this pattern. Lets look at how this can be accomplished.

First, we would create a new class that will be serialized as the message body of the BrokeredMessage. In this case we will create a class for membership.

    [Serializable]
    public class Member
    {
        public Member()
        {
            
        }

        public bool NewMember { get; set; }
        public string Name { get; set; }
        public string Address { get; set; }

        [PromotedProperty("ReceiveEMail", PropertyDirection.ReadAndWrite)]   
        public string ReceiveEmailAds { get; set; }

        [PromotedProperty("Licensed", PropertyDirection.ReadOnly)]
        public bool Licensed
        {
            get { return (....); }
        }
    }

As you can see we have decorated a number of the properties with custom attributes. We decorate every property that we will want to add to the BrokeredMessage properties collection. In addition, we can set the PropertyDirection to either ReadOnly or ReadAndWrite. ReadOnly certainly makes sense – we just need to take a value of the property and promote that into the collection. However, we can take this beyond just static promotion. With the ReadAndWrite we can provide the functionality to update the property as the message is received from the Topic. With the Subscriptions on our Topics we have the ability to perform an action in the rules. In that action we can set the value of a property, and in this case, we can push that modification all that way back to the internal property on the object. We can then use that for other downstream actions in our processing of the message. This give us a lot more flexibility and functionality in our solution.

After we have our object decorated we can put together the code to send the message. We can take the same method that we used earlier to submit a message but we will do a couple of different things.

As you will notice, the code to submit the message is pretty much the same – we create the factory, create the message sender, create an instance of our member object and create a BrokeredMessage. In order to get our promoted properties code to work as an extension to the BrokereMessage we needed to utilize the Extension Methods functionality in C#. We needed to do this because the the BrokeredMessage class is sealed. One of the drawbacks of the Extension Methods functionality is that it doesn’t work on the objects constructor. Therefore, we need to call the PromoteProperties method that is placed on the BrokeredMessage class. By doing this, this method will interrogate all of decorated properties on your object and promote them.

public static void SubmitMessageToTopic()
 {
 String issuer = ....;
 String key = ....;

 TokenProvider tp = TokenProvider.CreateSharedSecretTokenProvider(issuer, key); 
 Uri uri = ServiceBusEnvironment.CreateServiceUri(.., .., ..);

 MessagingFactory factory = MessagingFactory.Create(uri, tp);
 MessageSender myTopic = factory.CreateMessageSender("<TopicName>");

 //Create our message body object

 Member memberItem = new Member();

 BrokeredMessage message = new BrokeredMessage(memberItem);

 //Now that we have created the message and passed in our object we need to do one more thing

 message.PromoteProperties(memberItem);

 //even thought we are promoting properties, we can still add properties at design time as usual as shown below

 //message.Properties.Add("MemberType", Gold);

 myTopic.Send(message);

One other item I want to mention in the code above is that even though you promote properties decorated in your object you can still add properties to the BrokeredMessage properties collection at design time as usual.

So, we now have our object decorated and we have the code to submit the message to the Topic. Lets look at the extension method that promotes the properties.

public static void PromoteProperties(this BrokeredMessage msg, object serializableObject)
        {
            PropertyInfo[] properties = serializableObject.GetType().GetProperties();

            foreach (PropertyInfo prop in properties)
            {
                foreach (PromotedProperty promotedProp in prop.GetCustomAttributes(typeof(PromotedProperty), true))
                {
                    object value = prop.GetValue(serializableObject, null);
                    msg.Properties.Add(promotedProp.Name, value);
                }
            }
        }

This method loops through the properties in the object, find the promoted properties and adds the value of the property with the name provided in the attribute to the BrokeredMessage’s property collection. There are other other methods in the extension class for things such as setting the properties value for our ReadAndWrite property direction but this gives an overview on how we can add this functionality and provide not only a great implementation of this pattern but also extend the functionality of the Azure Topics and routing functionality.

Recipient List

This pattern is related to the Content Based Router pattern in that the CBR pattern routes messages based on the message content. The Recipient List pattern provides a means for the message to contain a list of one or more recipients to receive the message. This differs from CBR in that we may want to submit a message to multiple order fulfillment suppliers in order to receive back prices thus allowing for the comparison of responses and the selection of the cheapest supplier as shown in the diagram below:

This pattern can be implemented in the Azure Service Bus Topic very similarly to that of the Content Based Router example. The difference will be that we will create the subscriptions using the SQLFilter but will utilize the LIKE keyword. This way the sender can populate the property with a comma separated list and the subscriptions will pick it up and route to N number of receivers.

So, if we created three subscriptions

topic.AddSubscription("OtherMemberTypes", new SqlFilter("MemberType LIKE ‘%Premium%’"));

topic.AddSubscription("OtherMemberTypes", new SqlFilter("MemberType LIKE ‘%Gold%’"));
topic.AddSubscription("OtherMemberTypes", new SqlFilter("MemberType LIKE ‘%Silver%’"));

and we wanted to create a message that would be routed to Premium and Gold we would create a BrokeredMessage and set the following property:

message.Properties.Add("MemberType", “Premium,Gold”);

This would allow us to set, at the client,  the recipients that we wanted to send the message to.

  Splitter

This pattern provides the ability to break apart a composite message into its separate message parts and process the related message parts separately. This pattern is useful in scenarios where you might have an order that has many order items and each of the order items need to be sent to different receivers or sent to different inventory systems.

This pattern can be implemented in Azure using the Service Bus Topic or Queue by utilizing the sessions feature. To setup a queue or topic for sessions we need to set the RequiresSession property in the QueueDescription or SubscriptionDescription. Then each message needs a SessionId in order to relate the individual messages. The receiver will read a message and accept the session. Once the session is accepted, the session is locked and other receivers won’t be able accept it. This ensures that there are no competing receivers. One thing to note about sessions and Azure is that session ordering is guaranteed even if the receiver crashes. Another thing to think about is this pattern comes in handy when you need to provide support for sending messages larger than the allowed message size. To send larger message, you can chunk the message into smaller sizes and use the session id in order to receive all the messages in the same session and then reassemble them.

There is a great example of this that Alan Smith has put together so I won’t duplicate the effort. Take a look at his blog for an example of implementing this pattern.

Aggregator

This pattern is related to the splitter pattern. Whereas the splitter processes separate messages, the aggregator is used to bring all the separate messages back together.

This pattern can be implemented in Azure using the Service Bus Topic and correlation. When setting up the subscription you can use the CorrelationFilterExpression just as we did with the FilterExpressions in the code samples shown above in previous patterns. So the submitter would set the CorrelationId property of the BrokeredMessage object to an identifier. Then this same identifier would be used in the CorrelationFilter object that was configured when creating the subscription. Also, you will want to know when the processing is finished for this correlation set. You could easily create a custom property on the BrokeredMessage object (as we did previously with the MemberType property) that lists the total number of messages that are in the set or you may just decide to create a flag that states that this is the last message in the set. Once the total count has been received or the message containing the flag is received then you can perform whatever task is required to aggregate all the messages together.

You might ask, why use Correlation over Session. The CorrelationId property provides an exact match, the CorrelationId is saved in a hash table and the CorrelationId processing is better optimized for matching performance than the SQL expression filters.

There is a great example of this that Alan Smith has put together that is in the same article as his Splitter implementation so I won’t duplicate the effort. Take a look at his blog for an example of implementing this pattern.

Resequencer

The Resequencer pattern provides guidance around putting messages back in order. Individual messages may be processed by a downstream system and returned faster than other downstream systems can process their messages and thus leading to messages that are out of order. This pattern uses a buffer to store messages so that they can be resequenced. Typically the Resequencer does not modify the message in any way. Once in the required order, the messages can be delivered as shown in the following diagram.

This pattern can be implemented in Azure through a Service Bus Queue. The queue becomes the buffer to not only store the messages that are submitted but also to store the messages that are out of order. There are a number of steps that will be required to put this pattern in practice. First, there is no change to the code that is used to submit the messages to the queue however you will need to populate the MessageId property of the BrokeredMessage object. While there is nothing out of the ordinary to submit, there is a bit of work to be done by the receiver. Next, we are going to take advantage of the Defer() method which will allow us to put a message back on to the queue that we detect is in the wrong sequence. Then we can dequeue it later when we need that message by calling the receive message and passing in the message sequence. This means that we need to keep a list of message sequence numbers in our code. One thing to note is that once we defer a message and put it back on the queue the only way that we can retrieve it again is to call the receive and pass in the sequence number. Another thing to note is that the Service Bus Sequencing may be different than the applications sequence numbering. Therefore you may need to store both in either session state or an internal list.

Lets look at the code to put this pattern into place.

First, let’s look at what is needed to send the message to the queue.

 string issuer = "....";
 string key = "....";

 //create the token provider and URI that are needed to send to the MessagingFactory
 TokenProvider tp = TokenProvider.CreateSharedSecretTokenProvider(issuer, key); 
 Uri uri = ServiceBusEnvironment.CreateServiceUri("sb", "<ServiceNamespace>", string.Empty);

 //retrieve the MessageSender for the SeqQueue within the namespace
 MessagingFactory factory = MessagingFactory.Create(uri, tp);
 MessageSender SeqQueue = factory.CreateMessageSender("SeqQueue");

 //create a message and use a string for the body.  We will put the sequence here as well as in a property.
 BrokeredMessage message1 = new BrokeredMessage("MessageSequence 1”);

 //set some additional custom app-specific properties
 message.Properties["MessageNumber"] = "1";
 message.Properties["TotalMsgsInSequence"] = "3";   
 //send the first message to the queue
 SeqQueue.Send(message1);

//create the next message in the sequence. This message will be out of order.
 BrokeredMessage message3 = new BrokeredMessage("MessageSequence 3”);
 message.Properties["MessageNumber"] = "3";
 message.Properties["TotalMsgsInSequence"] = "3";   
 //send the first message to the queue
 SeqQueue.Send(message3);

//create the last message in the sequence.
 BrokeredMessage message2 = new BrokeredMessage("MessageSequence 2”);
 message.Properties["MessageNumber"] = "2";
 message.Properties["TotalMsgsInSequence"] = "3";   
 //send the first message to the queue
 SeqQueue.Send(message2);

Now we have three messages in our queue that are out of order. Next, let’s look at what is needed to receive the message and process them in order. By the way, I will be utilizing the QueueClient object as this supports more advanced features (including sessions).

private static void ReadMessagesFromQueue()

....

// Read messages from queue until queue is empty: 
 
MessageReceiver Qreceiver = queueClient.CreateReceiver(); 
List<MessageReceipt> MessageReceiptsDeferred = new List<MessageReceipt>(); 
 
BrokeredMessage receivedMessage; 
while (Qreceiver.TryReceive(TimeSpan.FromSeconds(10), out receivedMessage)) 
{

  msgCount++;
  //check the received message and determine if it is in the sequence we need 
  if (receivedMessage.Properties["MessageNumber"].ToString() == msgCount) 
  { 
     
     ProcessEachMessage(receivedMessage); 
  } 
  else 
  { 
     receivedMessage.Defer();

     //store the message receipt for later 
     MessageReceiptsDeferred.Add(receivedMessage.MessageReceipt); 
  } 
} 
 
 
//we have now processed messages 1 and 2.  We need to go back and get message 3. 
foreach (MessageReceipt receipt in MessageReceiptsDeferred) 
{ 
      ProcessEachMessage(Qreceiver.Receive(receipt)); 
} 
 
 
Qreceiver.Close();

private static void ProcessEachMessage(BrokeredMessage message) 
{ 
         //This is where we do the work we need to on the messages (i.e., reassemble, send off to a LOB system, etc.)

         //Do your work here

       
         message.Complete(); 
}

This example shows you how we can put this pattern into code. As with all code examples, there is still work that needs to be done to this code if you are going to use this as a sample.

As you think about putting this pattern into practice, it could be useful for situations in which you want to process messages that have high and low priority all on one queue. You may not know what order the messages are submitted but you can receive all the messages, process the ones marked as high priority and defer the other submitted messages and then go back and process those messages after all the high priority messages have been drained.

Composed Msg. Processor

This pattern provides guidance for message processing where there are a number of sub messages (i.e., order line items) that need to be routed to individual destinations, processed, returned and aggregated back into a single message. This pattern is composed of a number of other patterns that include the Splitter, Content Based Router and Aggregator.

Because this pattern utilizes patterns already covered in this article I won’t duplicate the code as that can be found in each patterns implementation. However, I will include a diagram that shows how these patterns can come together to create this complex pattern.

Scatter-Gather

This pattern outlines how a single message can be sent to many receivers, or a message may be split and its parts sent to many receivers so that one of many external providers may process the message. Once the message has been processed and returned then the messages need to be aggregated and brought back into one message that will be processed. Think of an order that could be fulfilled by a number of suppliers. You can scatter the messages to those suppliers and gather all the responses and decide if you are going to take the first message returned and continue processing or if you analyze all the responses and take the lowest cost supplier. Either way, once you have received all the messages you will take the message and send it on to the next step in the process.

To put this pattern in practice we can take implementations of other patterns. The scatter portion can be implemented through the use of the Splitter pattern whereas the Gather portion can be implemented through the use of the Aggregator pattern. Both of which are covered above.

Process Manager

The process manager pattern outlines how you send a message through a number of processing steps that aren’t known at design time. These steps may not even be sequential. In addition, the sequence of the processing steps may also be modified based on the results of the previously executed processing step.

To put this pattern into practice you can utilized Windows Workflow. There are a number of blog articles showing how you can implement the currently released version of WF 4 in your code and deploy it to Azure. In addition, there was a CTP of WF in Azure that was released last summer. Lastly, there is also the Workflow Activity Pack for Windows Azure that allows you to run WF on-premises and interact with artifacts in Azure.

Integration Patterns utilizing the Windows Azure Service Bus

2012-04-16T16:36:56Z

For years we have all utilized Gregor Hohpe’s Enterprise Integration Patterns book (and the web site). Many of these patterns, if not most, are well known and there are many resources out there that talk about how these patterns can be implemented. There are also a number of resources that talk about how these patterns can be implemented in BizTalk or Windows Server AppFabric.

However, there is very little that shows how these patterns can be implemented using the Windows Azure Service Bus. This two part series will take a patterns first look at how these different patterns can be implemented.

The patterns that will be covered in this post will focus on the Messaging Channel groupings. The patterns in that fall in the Messaging Channel group contain:

Point to Point, Publish/Subscribe, Datatype Channel, Invalid Message Channel, Dead Letter Channel, Guaranteed Delivery, Channel Adapter, Messaging Bridge and Message Bus.

In the next post we will focus on the patterns that fall in the Message Routing group, which contain:

Content Based Router, Message Filter, Recipient List, Splitter, Aggregator, Resequencer, Composed Message Processor, Scatter-Gather, Process Manager and Message Broker.

There is also the Messaging Systems group of patterns but those are being addressed with the Azure Integration Components (that is currently in CTP). As those components get closer to release I will cover those patterns as well.

Lets start out by looking at each of the patterns in the Messaging Channel Group including a description of the pattern and then how it can be implemented.

Point-to-Point Channel requires that only one receiver gets the message. If there are multiple receivers, then only one must still get the message. This functionality can be done directly through WCF (the Service Bus can also be included by hosting the WCF service). This pattern can also be accomplished through the use of the Service Bus Queue and controlling what services are listening.

Publish-Subscribe Channel requires that the message or event sent to the channel will be delivered to each and every receiver. Once the receiver obtains a copy of the message it is removed from the channel. This can be implemented through the use of Service Topics.

A Service Bus Topic can have multiple subscriptions with each subscription being able to be individually managed. The subscriber receive all messages, based on filters, that are sent once the subscriber is created. If a message satisfies the filter criteria for more than one subscriber the message will be delivered to each subscriber and then onto the receiver. When a message needs to be delivered to more than one subscriber, the message is not duplicated. The meta data for the filter/subscriptions is stored and is tracked based on which subscriber/receiver has received the message. In terms of removing it from the channel, think of the concept of ref counts. One other thing to keep in mind is that if the Receiver is not online to receive the message, then it is stored until the receiver comes on-line.

Datatype Channel requires that each channel will contain the same type of data. Thereby when the receiver obtains a message from the channel it will always be the same type. This can be accomplished with directly with WCF (the Service Bus can also be included by hosting the WCF service).

In addition, an Azure Queue can also be utilized. You can create a new queue for each type of message that will flow through the queue.

Invalid Message Channel requires that a message that is deemed to be invalid be moved to a Invalid Message Channel where they will not be processed by the normal receiver.

The Azure Queue implements the dead letter queue concept and uses the $DeadLetterQueue subqueue to store invalid messages. Messages are placed in the dead letter subqueue:

When a message expires and deadlettering for expired messages is set to true in a queue or subscription.
When the max delivery count for a message is exceeded on a queue or subscription.
When a filter evaluation exception occurs in a subscription and deadlettering is enabled on filter evaluation exceptions

To get to the dead letter queue you would use:

string queueDeadletterPath = QueueClient.FormatDeadLetterPath("<your queue name>");

the same behavior is available with Topics. However, to get to the dead letter queue for a topic you would use:

string subscriptionDeadletterPath = SubscriptionClient.FormatDeadLetterPath("<Your Topic Name>", "<Subscription Name>");

Dead Letter Channel requires that if the system can’t deliver a message there is a mechanism to move the message to a dead letter channel. This capability is handled like the Invalid Message Channel above. However, check out this link from MSDN on the QueuePolicy.PoisonMessageDrop property. It points to the prerelease documentation but it is interesting to read what is being planned.

Guaranteed Delivery requires that the system persists the message so that it is not lost if there is a crash or outage. In using the Service Bus Queues, you are ensured of guaranteed delivery as the queue uses SQL Azure as the backing store.

Channel Adapter requires that there is an interface so that any application can connect and be integrated with other applications. The Service Bus Queues and Topics both provide a full API that can be utilized by a number of different technologies.

Messaging Bridge provides a connection between different messaging systems. Essentially to transmit message that was intended for another messaging system. This can be done through either the Service Bus Queues or Topics with the receiver getting the message and forwarding it to endpoints provided by the other messaging system. In addition, the receiver can get the message and forward it on to on-premise messaging systems through WCF.

Message Bus provides a means for separate application to work together in a decoupled manner with the ability for the applications to be added or removed with no affect on any other applications. The ability to be decoupled but still allowing the application to work together, share data and operate in a unified manner is the basis for this pattern. This pattern can be easily implemented through the Azure Service Bus and WCF endpoints. We will come back to this pattern again when we spend more time covering the integration components.

In this post I have talked about the Service Bus Queue, but there is still some confusion between the Windows Azure Queue and the Service Bus Queue. Yes, there are two Queues and they are different and have different uses and purposes. Everything I have been referring to here is in reference to the Service Bus Queue. You can find more information on the differences between Azure Queues and Azure Service Bus Queues in this article - Windows Azure Queues and Windows Azure Service Bus Queues - Compared and Contrasted.

Lastly, in talking about Gregor Hohpe and his patterns book, there are a number of diagrams that are utilized that represent the patterns. These diagrams can be downloaded and used in Visio which you can find at http://www.eaipatterns.com/downloads.html

Services Hosted in Windows Azure Service Bus

2012-03-20T19:50:45Z

In my last post, I talked about the BizTalk AppFabric Connect for Services functionality for BizTalk. As you start to make your services accessible through the cloud you may find that you hit an error when you try to access the endpoint. You might also find that your endpoint doesn’t show up in the Windows Azure Portal.

In my case, I saw this error, in the screen shot below, when I clicked on the URL for my service in the AppFabric portal.

Then I saw that I had the following error in the Event Log.

I knew why the error in the Event Log was showing up. I have Windows Server AppFabric installed and had configured my service to Auto-Start. What I didn’t know was why I was getting the underlying error of not being able to reach watchdog.servicebus.windows.net. After doing a lot of searching on-line I couldn’t find anything that could solve the problem.

So, after talking to a member of the dev team who created the wizard functionality we found a fix. It turns out you will get this error when the service can not communicate with Azure. Since I had a proxy server in the mix, the fix was to add an entry in the config file in order for communications to flow through the proxy server. When the wizard completed it created a set of artifacts in IIS including a config file. Open that config file and we will add the entry below and place it just before the system.web section.

...
<connectionString />

<system.net>
   <defaultProxy useDefaultCredentials="true">
      <proxy proxyaddress="[your proxy address here]" />
   </defaultProxy>
</system.net>

<system.web>
...

Enter your proxy address and you should see this error go away.

BizTalk AppFabric Connect For Services

2012-03-20T19:36:10Z

Last week I spoke at the Twin Cities Connected Systems User Group Meeting. The title of my presentation was Integration with Windows Azure – Connecting to the Cloud. During my presentation I talked about ways to expand on-premise applications to the cloud as well as how to connect cloud applications back to on-premise resources. As part of my presentation I talked about the functionality provided through the BizTalk AppFabric Connect for Services. There were a number of people that hadn’t heard of it so this entry talks about what it is and how it can be used.

So, as you build your composite applications, or just look to extend your current application, you may want to take advantage of the benefits offered by extending your application to Windows Azure. This will let you leverage your existing on-premise artifacts, and LOB systems, in order to expose them and extend their reach to external clients in a secure manner.

Microsoft released the BizTalk Server 2010 AppFabric Connect for Services as a new feature for BizTalk 2010 which provides functionality to expose your BizTalk artifacts (schemas and Orchestrations) as WCF Services exposed through the Windows Azure cloud. This new functionality is exposed through changes to the BizTalk WCF Service Publishing Wizard. When you run this wizard you will see a number of new screens in addition to the screens that you are already familiar with.

To expose BizTalk services to Windows Azure, specifically Windows Azure ServiceBus, you will need to have a few prerequisite components installed.

First, of course, you must have BizTalk 2010 and the BizTalk development tools installed on the machine where the AppFabric Connect for Service feature will be used.

Next, you must have the Windows Azure AppFabric SDK (1.0 or later) installed.

Lastly, there are benefits of having Windows Server AppFabric installed. While this is not a requirement, it makes configuration setting easier (such as Auto-Start).

After all the pre-requisites have been installed, you will need a Windows Azure account and you will need to create a namespace for the Service Bus in the Azure Portal.

Once this is done then you can start the wizard and walk through each of the screens. The wizard walks you through selecting the type of endpoint (service or metadata) – as it did previously. The next screen will be a new screen which will fork between the functionality of the old wizard (exposing web services on-premise only) or to sprinkle in the new screens to allow you to expose the services to the cloud. This screen presents a single checkbox asking if you want to add a Service Bus endpoint. It also reminds you that you need to have a ServiceBus service namespace. Select the checkbox (otherwise there is really no need to keep reading this post). The next five screens are the same as the screen in the original wizard (select either schemas or Orchestrations, select the assembly, select the public ports, assign the target namespace (and remember to check allow anonymous access)). One thing that I found very helpful is that the wizard will take all of the selected ports and merge them into a single WCF service. The last two screens are specific to the configuration of the Service Bus.

The first is the Endpoint Configuration.

There are a number of options that will effect the output of the wizard. The first is the Relay Binding option and you have three choices – BasicHTTPRelayBinding, NetTcpRelayBinding, and the WS2007HttpRelayBinding. The Service Namespace is the namespace that you created in the Azure portal. I am sure that this goes without saying but what you enter here must match with the name you created in the Portal.

The Enable Discover checkbox lets the services you are exposing be publicly discoverable through the ATOM feed page created by Azure for your Azure Service Bus account.

Lastly, the Enable metadata exchange on cloud tells the wizard to create the metadata endpoint in the cloud so that external clients can generate a proxy for your service.

The next screen takes the authentication information that you can find on the portal site.

You will need to enter the Issuer Name and Key. When you created the namespace in the Azure ServiceBus the issuer name defaults to owner. Enter both the Name and Key into the text boxes. The first checkbox controls whether the consumer needs to authenticate using to the relay service to consume the endpoint. The second checkbox controls the same but for the metadata endpoint. Since you are charged per access to your endpoints you may want to set these to prevent unauthorized access.

Click Next and Finish to complete the wizard. Once it is finished you will have a new Receive Port/Location and a site in IIS. However, there are still a number of important steps that need to be done in order for your service to work.

If you created a service based on an Orchestration then you need to bind the port to the orchestration in the BizTalk Administrator. You will also need to Enable the Receive Location and Enlist the Orchestration.

Next we move to IIS. We need to have an Application Pool that is configured to use the .NET Framework 4.0 with the Pipeline set to Integrated. Set the site to use the Application Pool by right clicking on the site that was created. In the popup menu, select Manage Application and then click on Advanced Settings. When the Advanced settings dialog box appears change the Application Pool to use the one you just created.

Lastly, we need to start our service. If you are running IIS 7.5 (Windows 7 or Windows Server 2008 R2) then you can configure Auto-Start. If you are running a different OS version then you will need to start the services using the on-premise endpoint with a local client. Once the service is started it will make contact with Windows Azure and register the service. One thing to note is that this is the first time in the entire process that your service has communicated with Azure.

If you have Windows Server AppFabric installed then configuring Auto-Start if very simple. To do this, right click on your site and select Manage WCF and WF Services in the popup menu, then click the configure item in the fly out menu. When the dialog box appears, select Auto-Start from the left hand side and then click the Enabled radio button.

You are now ready to test your endpoint. Point your browser to .servicebus.windows.net">.servicebus.windows.net">http://<servicebusnamespace>.servicebus.windows.net. You should see the ATOM feed page and your service should be listed. You can click on the service and you should see a new page with your service and MEX endpoint. You can now consume this endpoint the same is if it were on-premise.

Can you move a Virtual Machine from VirtualPC to Hyper-V?

2012-01-19T00:31:43Z

I was recently asked if I could help move a virtual machine that had been setup and running in VirtualPC and move it to run under Hyper-V.

The answer is yes it can be done. Movement from Hyper-V to VirtualPC is not as easy (and many times not possible) but there are a number of blog posts on this topic already so I won’t cover that here. One of the reasons that it is easier to move from VirtualPC to Hyper-V is that VirtualPC is only 32 bit.

There are a number of steps that must be occur for a successful move.

First, start by uninstalling the integration components while the virtual machine is running in VirtualPC. You can do this through the Add/Remove programs feature in Windows in the Virtual Machine. The Hyper-V drivers and additions will not install over the VirtualPC additions and that is why you must remove them first.

Next, move the vhd file to a location where it can be accessed by your Hyper-V instance. Walk through the wizard to create a new virtual machine but when prompted to create a new drive or select and existing drive, pick select an existing drive and point it to your .vhd file.

Finally, once you have the virtual machine configured in your Hyper-V instance then start the machine. Go through the Settings Menu and install the Hyper-V additions. Once you do this, the Hyper-V additions installs a new HAL as well as new drivers for network, video and sound devices. The process of installing the new HAL is one of the reasons that a Hyper-V image is no longer portable back to VirtualPC.

However, at this point, you might think that everything is done and you are ready to use the virtual machine. Most of the time this is correct, however, there are situations that require additional steps. You will know that you have additional steps if your integration components aren’t working – you can tell really quickly if your mouse doesn’t move outside of the virtual machine.

You are more likely to have this occur if your virtual machine is running versions of Windows prior to Vista or if you are running Windows Server 2008 as these do not have the ability to dynamically detect the HAL at boot time.

So, to fix this run MSConfig.exe – by clicking the Start menu, selecting Run and typing msconfig. Once the utility launches, click on the Boot tab and click the Advanced Options Button. When the BOOT Advanced Options dialog appears, click the Detect HAL checkbox and hit ok. Restart the virtual machine and you should be good to go!

Composite Applications

2011-03-16T20:49:09Z

It has been a long time since my last blog post. It certainly hasn’t been for a lack of content but it is amazing how fast time flies when you are busy. I have been spending a lot of time lately working on composite applications and will be posting a number of entries around this topic.

One of the first questions that people ask is what is a composite app?

Composite apps can take on many forms and use many differing technologies. Many times people will talk about composite Apps and will only be referring to the UI tier. While the concept of a UI composite can be compared to a mashup (a UI that combines publically available web based services to create a meaningful combination of information to a user) a UI composite is built to use business sources – in many scenarios these sources (assets) already exist within the organization from previous application development efforts.

A composite app can also refer to just the middle tier as well. Many companies have spend a lot of time and money over the past many years building out SOA services. A composite app can consist of orchestration services that draw on the functionality of these different services. One thing to keep in mind however, is that a composite app is not make a SOA architecture. Instead they they will build on top of the work that has already been done to create a SOA architecture.

The idea of a composite app is to build upon existing software assets to quickly bring data to consumers in new way.

Many of the articles and blogs on this subject talk a lot around the UI tier. While that is a very important part of a composite application, I am going to spend time over the next few posts talking about composites at the middle tier.

There is a very good all-up article titled What Are Composite Applications from Atanu Banerjee where he walks through the full composite application stack. In this article he outlines three steps that a solution architect must do to design a composite application. Those things are to:

Choose a composition stack. Pick one or more containers from each tier, and a set of component types that are deployable into those containers.
Choose components. Define the repository of assets that must be built from this set of component types, based on business needs.
Specify the composite application. Define the ways in which those assets will be connected, to provide a particular cross-functional process. The platform should enable these connections to be loosely coupled.

Each of these things are required so that the consumer can get the most benefit. Over and above this, the organization should start to benefit by realizing faster time-to-benefit turnaround as well as faster time-to-deployment. These two, along with exposing existing LOB data are the three driving factors that are making the idea of composite applications so popular.

In my next several posts, I will cover some of the technical choices, how to bring bridge between services that live in BizTalk and services that live in AppFabric, and some of the lessons learned.

Twin Cities Connected Systems User Group Meeting–March 17th, 2011

2011-03-16T15:39:29Z

If you are in Minneapolis on Thursday March 17th please join us for the Twin Cities Connected Systems User Group Meeting.

The meeting takes place at 6:00 p.m. at the Microsoft offices at 8300 Norman Center Drive, Bloomington, MN 55437.

Andy Morrison will be presenting on a Lap Around BizTalk 2010 and the ESB Toolkit 2.1

Twin Cities Connected Systems User Group Meeting – September 16th, 2010

2010-09-10T17:16:02Z

If you are in Minneapolis on Thursday September 16th please join us for the Twin Cities Connected Systems User Group Meeting.

The meeting takes place at 6:00 p.m. at the Microsoft offices at 8300 Norman Center Drive, Bloomington, MN 55437.

Eric Kraus and I will be speaking on: Composite Applications: Connecting and Building Workflow Applications

Here is the write-up:

Come learn about Composite Applications and how you can build chunks of functionality in different loosely-coupled technologies that can be used to make up a larger collective solution. We will discuss how you can use SharePoint, BizTalk, Windows Server AppFabric, .NET and the LOB Adapters to build these chunks of functionality as well as the applications that consume them.

Storing Application Configuration Information in SSO

2010-08-31T15:59:12Z

For those using BizTalk, we have long had the need to store configuration information in a central location that is shared between each of the distributed BizTalk instances. BizTalk itself uses SSO for all of its internal configuration information. Unfortunately, we have not had a tool that allows us to take advantage of this functionality. Currently, in SSO there are three different utilities to perform SSO based tasks. These are the SSOConfig, SSOManage and the SSOClient utilities. All of these tools focus on managing credentials.

There has been a lack of tooling for the ability to create and manage configuration based applications. This has now changed with the creation of the SSO Configuration MMC Snap-In. This tool is an MMC based utility that allows developers to store name/value pairs in the the SSO configuration database. The tool can be downloaded from the Microsoft downloads site and includes three parts.

These are the MMC management tool, a C# class file that provides the code necessary to access and read the configuration information and lastly, an automated MSBuild task to use for deployment of your configuration information.

The MMC Snap-In provides an easy to use interface that allows you to add and manage applications, add and manage key value pairs as well as importing and exporting (in an encrypted format) configuration applications so that they can be deployed to different environments.

The C# client class can be used in your application or helper assembly to read the values stored in SSO.

And lastly, there is the MSBuild custom task that can be used to automate the deployment of your applications SSO configuration data. All that you need to do is export the application and use the MSBuild task to automatically import your settings into your new environment.

After all of this you might be asking “Why would you want to store your configuration data in SSO?”. Well, middle tier server based applications have the need to utilize configuration data that needs to be able to be read at runtime and changed without compiling. These applications also need the ability to have centrally managed and non repeated configuration information be available to all hosts in the middle tier. Typically, middle tier applications implement a custom solution and the architects/developers need to create a means to replicate this information to servers when a scale out solutions are implemented. Instead, this configuration can be stored directly in SSO and since SSO is already accessible to all servers in a scale out solution, we can take advantage of tools that we already use instead of creating custom solutions.

Using The Windows Server AppFabric Cache with ASP.NET

2010-08-30T14:53:42Z

It appears that I have lost some of my blog posts. I am reposting this blog post which originally appeared back in May.

Did you know that you can use the AppFabric Cache with ASP.NET? AppFabric Cache provides an ASP.NET session state provider. There are a number of reasons that you would want to consider using the AppFabric Cache instead of other caching technologies, including the built in ASP.NET caching.

The AppFabric Cache provides a number of benefits to ASP.NET programmers. When web applications need to maintain state, especially across a Web Farm, or needs to maintain objects across restarts AppFabric Cache provides the out of process, distributed highly available functionality you require. In addition, because of the distributed nature of the cache you no longer need to worry about sticky routing (getting the client back to the same machine as their cached objects). You also no longer need to add additional code and incur the overhead of storing state in a database. By keeping activity oriented objects closer to the consuming logic you can eliminate the overhead and increase performance of your application.

To utilize the AppFabric Cache ASP.NET session state provider open your applications' Web.Config file and enter the following XML fragment.

<system.web>
    <sessionState mode="Custom" customProvider="SessionStoreProvider">
        <providers>
            <add name="SessionStoreProvider" type="Microsoft.ApplicationServer.Caching.DataCacheSessionStoreProvider,  Microsoft.ApplicationServer.Caching.Client, Version=1.0.0.0, Culture=neutral,  PublicKeyToken=31bf3856ad364e35" />
        </providers>
    </sessionState>
</system.web>

Tagging Objects in the AppFabric Cache

2010-04-23T03:02:00Z

In two of my previous entries I outlined functionality and patterns used in the AppFabric Cache. In this entry I wanted to expand and look at another area of functionality that people have come to expect when working with cache technology.

This expectation is the ability to tag content with more information than just the key. As you start to examine this expectation you will soon find yourself asking if the tagged data can be related to each other and finally if it is possible to remove the related data.

One of the nice things about the AppFabric Cache is that it is so easy to work with and can be so simple. Then, if needed, like with this topic, you can take advantage of the addition functionality. This is where the real benefits of AppFabric Cache come into play over the other cache alternatives.

Lets take a deeper look at tagging. Tagging is essentially assigning one or more string values to an object contained in the cache. The object in the cache can then be retrieved by using the tag or multiple tags. When you tag an object, all of the tagged objects will reside within a region. A region is an additional container inside the cache. Because regions are optional, if you want to use them you need to explicitly create them at runtime within your application code. You can setup different regions for different types of related data. When you utilize regions, you have the option to retrieve all objects in that region and therefore by default have setup a solution that now provides you access to all the dependent data. We can take this one step further in that you can also delete all of the data within a region and not have to worry about looping through all entries to find the specific keys and their dependencies.

As an example we could create a region called 'Beverages' and then we could cache all of the beverages we sell and tag each item by the type of beverage such as Soft Drink, Wine, Beer. We could even go one step further and provide multiple tags so that we could further segment the Wine category into White or Red or Merlot, Zinfandel, Riesling, etc. At this point the application could retrieve all of the catalog items based on the search criteria that were entered.

Lets look at how we can setup the region and tags. We will also look at the methods that are available to interact with objects using tags and lastly, how we can manage the data in the cache; adding, retrieving and deleting.

Setting Up a Region:

The code that follows assumes that you already now how to create a cache (either through code or through the PowerShell cmdlets (as found in my previous post here)).

To create a region we already need to have the cache created. Once that is done, we can pass in the cache name, use the GetCache method and then call the CreateRegion method passing in a region name. We can create as many regions as we need based on the manner in which you wish to segment the data. One thing to keep in mind however is that there can be performance implications when using regions. The code below shows how we can create the region.

private void CreateRegion(string CacheName, string RegionName)
{

    //This can also be kept in a config file
    var config = new DataCacheFactoryConfiguration();
    config.Servers = new List<DataCacheServerEndpoint>
        {
            new DataCacheServerEndpoint(Environment.MachineName, 22233)
        };

    DataCacheFactory dcf = new DataCacheFactory(config);

    if (dcf != null)
    {
        var dataCache = dcf.GetCache(CacheName);
        dataCache.CreateRegion(RegionName);
    }
}

Now that we have a cache with a region created lets look at the methods that are available to interact with tags

Methods to work with tagging (from MSDN):

Method	Description
GetObjectsByTag	Provides a simple way to access objects that contain tags (exact match, intersection, or union). The region name is a required parameter.
GetObjectsByAnyTag	Returns a list of objects that have tags matching any of the tags provided in the parameter of this method.
GetObjectsByAllTags	Returns a list of objects that have tags matching all of the tags provided in the parameter of this method.
GetObjectsInRegion	Returns a list of all objects in a region. This method is useful when you do not know all the tags used in the region.
GetCacheItem	Returns a DataCacheItem object. In addition to the cached object and other information associated with the cached object, the DataCacheItem object also includes the associated tags.
Add	When adding an object to cache, this method supports associating tags with that item in the cache.
Put	When putting an object into cache, this method can be used to replace tags associated with a cached object.
Remove	This method deletes the cached object and any associated tags.

Managing the data in the cache

There are a number of methods to get objects out of the cache but logically I like to start in terms of adding to the cache first. So, lets take a look at the put method. The put method will update an object that already has a key that is contained in the cache (whereas the Add will return an exception if the key is already present). The put method can also update or add new tags to an existing object in the cache. As we look at the Put method signature below we can see that this version of the method accepts the key and value just as the other overrides of the Put method do but this one also adds on a collection of tags as well as the name of the region that the cached item will reside in.

public DataCacheItemVersion Put (
string key,
Object value,
IEnumerable<DataCacheTag> tags,
string region
)

The code below shows how we can use the Put method and include multiple tags

private void InsertCacheObjectWithTag(string CacheName, string RegionName)
{
    //This can also be kept in a config file

    var config = new DataCacheFactoryConfiguration();
    config.Servers = new List<DataCacheServerEndpoint>
    {
        new DataCacheServerEndpoint(Environment.MachineName, 22233)
    };

    DataCacheFactory dcf = new DataCacheFactory(config);

    if (dcf != null)
    {
        List<DataCacheTag> tags = new List<DataCacheTag>
            {
                new DataCacheTag("Wine"),
                new DataCacheTag("Red"),
                new DataCacheTag("Merlot")
            };

        var dataCache = dcf.GetCache(CacheName);

        dataCache.Put("WineKey", "WineValue", tags, RegionName);
    }
}

Now we can look at the methods to retrieve objects. There are four main Get methods. We can get by tag, any tag, all tags and finally any object that exists in the region no matter what tag. The GetObjectsInRegion method is what provides us the ability to implement a scenario in which all cached objects are related and can be treated as a group. The related data can also be removed by called dataCache.RemoveRegion(RegionName) just as we called the CreateRegion method above.

For people that have been working with Memcached you can delete dependent data through the cas (check and set) operation. The reason that I bring this up is that more people are familiar with Memcached and therefore I am asked if AppFabric Caching has this or that functionality. What I am finding is that it has the same functionality and more. It is just implemented a bit differently.

Anyways, if we want to retrieve all the objects in the cache that have a Wine tag we can use the GetObjectsByTag as shown below:

public IEnumerable<KeyValuePair<string, object>> GetLookUpCacheDataByTag(string TagValue)
{
    var config = new DataCacheFactoryConfiguration();
    config.Servers = new List<DataCacheServerEndpoint>
        {
            new DataCacheServerEndpoint(Environment.MachineName, 22233)
        };

    DataCacheFactory dcf = new DataCacheFactory(config);

    var cache = dcf.GetCache(cacheName);

    DataCacheTag dct = new DataCacheTag(TagValue);

    return cache.GetObjectsByTag(dct, regionName); //regionName could either be passed in or an internal variable
}

and lastly, we can remove items based on a tag. I already mentioned that we can call the RemoveRegion method to remove all the related data that is grouped together in a region. If there is a specific item that you want removed then you can call the Remove method and pass in the key. If you wanted to delete based on a tag value then you would have to call one of the GetXXX methods, obtain the key, and then call the Remove method by passing the returned key value.

Pre-Populate the AppFabric Cache

2010-04-07T19:22:00Z

When I start talking to people about the caching functionality that is part of Windows Server AppFabric I am usually asked "What is the AppFabric Cache?" The MSDN page at http://msdn.microsoft.com/en-us/library/ee790954.aspx provides a great overview (below) as well as additional information. The Cache is defined as:

"Windows Server AppFabric caching features use a cluster of servers that communicate with each other to form a single, unified application cache system. As a distributed cache system, all cache operations are abstracted to single point of reference, referred to as the cache cluster. In other words, your client applications can work with a single logical unit of cache in the cluster regardless of how many computers make up the cache cluster. "

When you develop against the AppFabric Cache you utilize the cache-aside pattern. This pattern outlines the steps in which the application will first check to see if the data is in the cache. If not, then query a database (or other data source), load the cache, then return the value. The idea is that the cache will be populated over time as instances of the application call for data. The AppFabric Cache comes with a number of options around how long data should remain in the cache and provides you the flexibility to tune the performance to your needs.

One thing that the cache-aside pattern doesn't provide for is the pre-population of data in the cache. This entry will walk through how we can provide this functionality.

The first thing we need to do is to create the cache. I outline how to create a cache in .NET code in my previous entry. You can check that out but I will also include the code in the sample below. As you go about creating the cache you also need to decide if you want eviction to occur and if so how long the data will stay in the cache until it is evicted. For this example I am going to setup the cache so that eviction is turned off (Expirable is set to false). Since I am preloading the cache I want to the data to remain in the cache. If I read your mind, the next question will be how will the code handle when new data is entered into the database. Through the use of the SQL Dependency functionality we can setup an event handler to respond when an event is raised when there is a change at the database layer. Once this event is caught we can empty the cache and reload it. We could also add code to take advantage of the cache-aside pattern and if we don't the data in the cache then query the database and populate the cache on demand.

As we look at the code example below lets jump directly to the PopulateLookUpCache method. The first part of this method sets up the database connection, the SQL Dependency and loads the data into a dataset. The second part of the method focuses on the loading of the cache. This is the part that we will focus on.

Before we can use the cache we need to create an instance of the DataCacheFactory. This object requires configuration information. This can either be through entries in a config file as shown below:

or through code by creating a DataCacheFactoryConfiguration object and passing it to the DataCacheFactory as shown below:

var config = new DataCacheFactoryConfiguration();
config.Servers = new List<DataCacheServerEndpoint>
{
new DataCacheServerEndpoint(Environment.MachineName, 22233)
};

DataCacheFactory dcf = new DataCacheFactory(config);

We can now call the GetCache method which will return a cache object based on the name of the cache that is passed into the method. The Cache object has a number of methods which allow us to interact with the cache. I use the Put method as the Put will add or replace an object in the cache whereas the Add method only adds an entry. If there is already an object in the cache it will return an exception.

In the code below, I loop through the DataSet and call the Put method to populate the cache. The Put method takes two parameters. The first is the key that will be used and the second is the value. In this case, since I am using the AdventureWorks sample database, the Products table contains the product id and the product which works very nicely for this example.

Once the loop has been executed and the data is now in the cache we can double check that the cache has been populated through the use of a PowerShell cmdlet. Switch to a PowerShell command window and type in (without the quotes) "Get-CacheStatistics -CacheName <your cache name>" and hit enter. This will return a list of attributes related to the cache and will show you how many items are currently in the cache.

We now have a populated cache and a now can use the GetLookUpCacheData method to read data out of the cache for our application. Take a look at the code below to see how all of this was done as well as how to setup the SQL Dependency code to get events. As always this code is provided as is and there are a number of places that should be refactored - such as the repeating DataCacheFactory objects.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Data.SqlClient;
using System.Data;

using Microsoft.ApplicationServer.Caching;

using System.Management.Automation;
using System.Management.Automation.Runspaces;

namespace AppFabricCacheWrapper
{
    public class CacheHelper : IDisposable
    {
        private static DataSet lookUpDataset = null;
        private string connString = string.Empty;
        private string cacheName = "PreLoadSampleCache";

        public CacheHelper()
        {
            connString = "Data Source=(local);Initial Catalog=AdventureWorksLT;Integrated Security=SSPI;";
            SqlDependency.Start(connString);
            CreateCache(cacheName);
            PopulateLookUpCache(cacheName);
        }

        ~CacheHelper()
        {
            SqlDependency.Stop(connString);
            Dispose(false);
        }

        public void Dispose()
        {
            Dispose(true);
            GC.SuppressFinalize(this);
        }

        protected virtual void Dispose(bool disposing)
        {
        }

        public string GetLookUpCacheData(string keyValue)
        {

            var config = new DataCacheFactoryConfiguration();
            config.Servers = new List<DataCacheServerEndpoint>
            {
                new DataCacheServerEndpoint(Environment.MachineName, 22233)
            };

            DataCacheFactory dcf = new DataCacheFactory(config);

            var cache = dcf.GetCache(cacheName);
            string data = cache.Get(keyValue) as string;
            if (data == null)
            {
                //Determine if you are going to query the database
            }

            return data;
            }

        /// <summary> /// Retrieves look up data for the given key and type from database /// </summary>
        private void PopulateLookUpCache(string CacheName)
        {
            SqlConnection conn = null;
            SqlCommand comm = null;
            SqlCommand commDependency = null;
            SqlDataAdapter sqlAdapter = null;

            //populate DataSet
            try
            {
                //Connect to look up database and retrieve the names of the products.
                conn = new SqlConnection(connString);
                conn.Open();

                comm = new SqlCommand();
                comm.Connection = conn;
                comm.CommandText = "select ProductID, Name from SalesLT.Product";
                comm.CommandType = CommandType.Text;

                if (lookUpDataset == null)
                {
                    lookUpDataset = new DataSet();
                }
                else
                {
                    lookUpDataset.Clear();
                }

                sqlAdapter = new SqlDataAdapter(comm);
                sqlAdapter.Fill(lookUpDataset);

                //Command object used for subscribing to notifications
                commDependency = new SqlCommand();
                commDependency.Connection = conn;
                commDependency.CommandText = "select ProductID, Name from SalesLT.Product";
                commDependency.Notification = null;
                SqlDependency dependency = new SqlDependency(commDependency);
                dependency.OnChange += new OnChangeEventHandler(dependency_OnChange);
                commDependency.ExecuteNonQuery();
            }
            catch (Exception e)
            {
                throw new Exception(e.Message + e.StackTrace);
            }
            finally
            {
                sqlAdapter.Dispose();
                comm.Dispose();
                commDependency.Dispose();
                conn.Close();
                conn.Dispose();
            }

            //populate cache
            try
            {
                //This can also be kept in a config file
                var config = new DataCacheFactoryConfiguration();
                config.Servers = new List<DataCacheServerEndpoint>
                    {
                        new DataCacheServerEndpoint(Environment.MachineName, 22233)
                    };

                DataCacheFactory dcf = new DataCacheFactory(config);

                if (dcf != null)
                {
                    var cache = dcf.GetCache(CacheName);

                    foreach (DataRow product in lookUpDataset.Tables[0].Rows)
                    {
                        cache.Put(product["ProductID"].ToString(), product["Name"].ToString());
                    }
                }
            }
            catch (Exception e)
            {
                throw new Exception
            }
        }
    }
    /// <summary>
    /// Event which will be fired when there are any database changes done to the dependency query set
    /// </summary>
    private void dependency_OnChange(object sender, SqlNotificationEventArgs e)
    {
        if (e.Info != SqlNotificationInfo.Invalid)
        {
            PopulateLookUpCache(cacheName);
        }
    }

    private void(string CacheName)
    {
        //This can also be kept in a config file
        var config = new DataCacheFactoryConfiguration();
        config.Servers = new List<DataCacheServerEndpoint>
            {
                new DataCacheServerEndpoint(Environment.MachineName, 22233)
            };

        DataCacheFactory dcf = new DataCacheFactory(config);

        if (dcf != null)
        {
            var state = InitialSessionState.CreateDefault();
            state.ImportPSModule(new string[] { "DistributedCacheAdministration", "DistributedCacheConfiguration" });
            state.ThrowOnRunspaceOpenError = true;
            var rs = RunspaceFactory.CreateRunspace(state);
            rs.Open();
            var pipe = rs.CreatePipeline();
            pipe.Commands.Add(new Command("Use-CacheCluster"));

            var cmd = new Command("New-Cache");
            cmd.Parameters.Add(new CommandParameter("Name", CacheName));
            cmd.Parameters.Add(new CommandParameter("Expirable", false));
            pipe.Commands.Add(cmd);
            var output = pipe.Invoke();
        }
    }
}

Interview on Endpoint.TV

2010-04-01T18:58:00Z

Danny Garber and I were interviewed for Endpoint.TV on Windows Server AppFabric and our upcoming book.

The interview has been published and is now available. Check it out at http://channel9.msdn.com/shows/Endpoint/endpointtv-Pro-Windows-Server-AppFabric/

Calling PowerShell from .NET

2010-03-31T20:08:00Z

I have been working with Windows Server AppFabric caching lately and have found it to be very impressive. The more that I work with it the more that I can see areas that it can be utilized. One of the things that will become quite evident as you start using it is that much of the setup and configuration is done through PowerShell cmdlets.

I am in the process of putting together an application and I want the application to be able to create and pre-populate the cache. As I looked into creating the cache I knew that it could be created through PowerShell directly but I wanted to call the PowerShell cmdlets from .NET. So, lets look at what is needed to create the cache from PowerShell first and then we will look at the code to call the cmdlets from .NET.

This assumes that AppFabric has been installed and configured.

Open a PowerShell command window. The first thing that we are going to do is to import two modules and then we can start issuing commands. First, type "Import-Module DistributedCacheAdministration" (without the quotes) and hit enter. This will import the cache administration cmdlets. Next type "Import-Module DistributedCacheConfiguration" and hit enter. This will import the cache configuration cmdlets.

The next command will bring the current machines' cluster configuration into the PowerShell session context. So, type in Use-CacheCluster and hit enter. We have now entered all that we need to start interacting with the cache. The first thing that we need to do is to ensure that the cache service is running. Type "Get-CacheHost" and hit enter. Look on the screen and see if the Service Status show UP. If it shows DOWN then issue the following commands to start it. Type "$hostinfo = Get-CacheHost" and hit enter. Then type in "Start-CacheHost $hostinfo.HostName $hostinfo.PortNo".

At this point we are ready to create the cache using the New-Cache cmdlet. Type "New-Cache <your cache name here>" and hit enter. This will create a cache with your name that is ready to use.

If you want to see the configuration for your newly created cache type in "Get-CacheConfig <your cache name here>" and hit enter. You will see the following configuration attributes and their settings.

CacheName : MyCache
TimeToLive : 10 mins
CacheType : Partitioned
Secondaries : 0
IsExpirable : False
EvictionType : LRU
NotificationsEnabled : False

To do this all with code we need to add three references to our project. Add the System.Management.Automation assembly as well as the Microsoft.ApplicationServer.Caching.Core and Microsoft.ApplicationServer.Caching.Client assemblies. These two assemblies can be found in the Windows\System32\AppFabric directory.

To set up the environment in .NET to issue PowerShell commands we need to start with the InitialSessionState class. This will create a session with the PowerShell runspace. A PowerShell runspace is the operating environment where you can create pipelines which will run the cmdlets.

So, just as above we will need to start with importing the DistributedCacheAdministration and DistributedCacheConfiguration modules. This can be seen below in the source code with the state.ImportPSModule method. Once we have the session state created we can call the RunspaceFactory to create a new runspace. At this point we can call the Open method can now start creating commands through a pipeline.

In the code below we create two pipelines. The first is our Use-CacheCluster command. This is a command that doesn't take or return any arguments. The next command is the New-Cache command. This takes a set of arguments so we will create this a bit different. First, we will create a Command object and then set parameters on the command object for each of the arguments we want to pass in. In this example I am providing a name as well as telling the caching system that I don't want the cache items to expire. Next, we need to pass the command object to the pipe and lastly, call the Invoke method as shown in the code below:

private void CreateCache(string cacheName)
{
    //This can also be kept in a config file
    var config = new DataCacheFactoryConfiguration();
    config.Servers = new List<DataCacheServerEndpoint>
        {
            new DataCacheServerEndpoint(Environment.MachineName, 22233)
        };

    DataCacheFactory dcf = new DataCacheFactory(config);

    if (dcf != null)
    {
        var state = InitialSessionState.CreateDefault();
        state.ImportPSModule(new string[] { "DistributedCacheAdministration", "DistributedCacheConfiguration" });
        state.ThrowOnRunspaceOpenError = true;
        var rs = RunspaceFactory.CreateRunspace(state);
        rs.Open();
        var pipe = rs.CreatePipeline();
        pipe.Commands.Add(new Command("Use-CacheCluster"));

        var cmd = new Command("New-Cache");
        cmd.Parameters.Add(new CommandParameter("Name", cacheName));
        cmd.Parameters.Add(new CommandParameter("Expirable", false));
        pipe.Commands.Add(cmd);

        var output = pipe.Invoke();
    }
}

We now have a cache created for us from .NET code.

Twin Cities Connected Systems User Group Meeting - March 11th, 2010

2010-03-10T15:08:00Z

If you are in are in Minneapolis on Thursday March 11th please join us for the Twin Cities Connected Systems User Group Meeting.

The meeting takes place at 6:00 p.m. at the Microsoft offices at 8300 Norman Center Drive, Bloomington, MN 55437.

I will be speaking on How to Create Windows Server AppFabric Applications

Here is a write-up of what will be covered:

You have heard about Dublin, now called Windows Server AppFabric, but do you know what it is and what it includes?

Do you know how to create WCF and WF applications/services that work better with AppFabric?

Come to the user group meeting and see how to write these applications and take advantage of AppFabric. We will show you how to use Visual Studio 2010 to create WCF and WF services and how to deploy, monitor, and control them with Windows Server AppFabric. In addition, we will also talk about what you need to know to start using AppFabric and what you need to do to optimize your existing applications and services to begin using AppFabric.

Pro Windows Server AppFabric Book

2010-01-18T22:40:00Z

I have been spending my free time working on my next book. Danny Garber and I are co-authoring this book from APress.

The book will be available this spring and should coincide with the release of AppFabric.

Twin Cities Connected Systems User Group Meeting - January 21st, 2010

2010-01-14T21:09:00Z

If you are in are in Minneapolis on Thursday January 21st please join us for the Twin Cities Connected Systems User Group Meeting.

The meeting takes place at the Microsoft offices at 8300 Norman Center Drive, Bloomington, MN 55437. This months meeting time has changed and we will be meeting from 5:00 to 6:30

Ed Jones from RBA will be presenting on Implementing a Service Bus Architecture with BizTalk 2009 and the BizTalk ESB Toolkit 2.0: A Real World Example

Here is a write-up of what will be covered:

Although BizTalk Server offers much in terms of flexibility and extensibility through the implementation of Service Oriented Architectures, most BizTalk applications are developed in “hub-and-spoke” models that are tightly coupled to specific points of functionality. Entire business processes are often represented as orchestration. As such, when business processes change, orchestrations also need to change, often requiring the reconstruction and redeployment of entire BizTalk solutions.

One way to alleviate this pain is to avoid the use of a “hub-and-spoke” model altogether in favor of a Service Bus approach. The BizTalk ESB Toolkit helps accomplish this by making the creation of a true Service Bus easier. One feature of the toolkit for example, itineraries, allows us to create capabilities that are independent of each other and independent of specific processes. This "Composition of Capabilities" method is preferred over the point-to-point solutions used in many BizTalk applications enabling more extensible and flexible Business Processes.

Our client required a system that would accept incoming shipment data in the form of flat-files in multiple formats, process that data through a series of resolutions, and then output the data in both its raw and processed form into an ERP system. Some data will be processed, while other data will be ignored. Over time it is expected that the various processes may change in size, scope, and sequential order. Our solution implements ESB Toolkit Itineraries to accomplish this composition of capabilities. We also use the Exception Management and other more traditional BizTalk functionalities such as Business Rules.

AppFabric and BizTalk

2009-11-24T05:37:00Z

There have been lots of questions lately about AppFabric (code named Dublin). These questions have centered around why Microsoft needs another middle tier solution and will AppFabric replace BizTalk.

Lets take a look at these questions as well as two additional questions; What is AppFabric and why does Microsoft need another middle tier solution? First, AppFabric is a distributed application server. To answer the other questions, we need to go in a little more depth.

Before .NET was released, developers working with the Microsoft technologies used COM+ to host their middle tier objects. Back then, when we needed to scale out our object oriented and object based applications we created middle tier code libraries and ‘hosted’ them in COM+. The COM+ host provided instance management (just-in-time activation), role-based security, automated transaction management as well as better memory and processor management, distributed transactions and a number of other services.

When .NET was introduced, we moved away from COM components and also moved away from using COM+ as a hosting model. This left a vacuum that so far had gone unfilled.

In attempts to fill this vacuum, individual developers had to create their own hosts. These hosts typically were Windows Services. However, these services usually didn’t include multi threading, scale out capabilities, tracking, monitoring, etc. – the types of functionality that has been in BizTalk for years. As Microsoft looked at BizTalk, as well as the .NET Framework, it was determined that there are many great features in BizTalk that if a scaled down version was available it would address gaps in the framework. Thus AppFabric was born.

For .NET developers writing applications using WCF and WF, AppFabric will fill the middle tier hosting vacuum. AppFabric will provide the host and will provide scalability and support for building out your middle tier application components. It also will simplify deployment, configuration, management and scalability of composite applications. The goal of AppFabric is to provide a server infrastructure, on which business logic developed in WF can be executed and exposed via WCF endpoints, without the need to design, develop, and support the infrastructural code.

The next question I hear is will AppFabric replace BizTalk? The answer is very clearly no.

AppFabric should be used when your architecture calls for an application level code-first object based approach. This is the same approach that is used by WF and WCF programming models and is the one that can be hosted in AppFabric and the additional functionality that AppFabric provides. AppFabric will provide a number of management and scalability features through IIS and WAS to provide the server infrastructure without the need to write the plumbing as you did before. This is a different approach than that of the XML Schema first approach that is used in BizTalk.

Use BizTalk when your architecture calls for an enterprise level message based approach. Use BizTalk when you need to isolate disparate systems that need to be connected. In point-to-point integrations, changes made to the provider system can have a profound impact on the consuming application. BizTalk natively provides a hub-based integration model which eases this burden and allows organizations to provide business services that are isolated from the impact of changes made to the systems and processes on which these services are based. This is achieved through the use of separate schemas, and the associated ability to easily develop message transformation logic using BizTalk’s Mapper tool. Use BizTalk if you have business-to-business requirements and need to integrate using technologies such as EDI, SWIFT, RosettaNet, AS2 or HL7.

The reality is that businesses will utilize both of these technologies in their different application architectures. This is not an either or decision but instead a decision of where and when to use each.

Keep in mind, however, that many of the same terms are used to describe the functionality provided with both BizTalk and AppFabric. These terms are things like Content Based Routing, Correlation, long running transactions, etc. Although the terms are the same the implementation and functionality is different. With all decisions, make sure that you take a couple of minutes to look under the hood and ensure that you understand the differences.

Follow Up from TechEd Europe

2009-11-21T05:46:00Z

Paolo Salvatori and I delivered a session titled 'Customizing and Extending the BizTalk WCF Adapters' and Paolo has created a number of fantastic blog entries around this subject. His latest entry goes through much of the content that was delivered in the session and he has included a number of links at the bottom for more WCF Adapter information and samples.

Speaking at TechEd Europe 2009

2009-10-13T05:02:00Z

I will be speaking at TechEd Europe 2009 in Berlin Germany November 9th through the 13th.

I will be presenting two sessions. My first is a 'Deep Dive with Microsoft BizTalk Server 2009 Development Platform' and my second is 'Customizing and Extending the BizTalk WCF Adapters'.

If you are going to be there stop by and say hello.

The Architecture Journal - Edition 21

2009-10-12T04:28:00Z

I co-authored an article titled 'Design Considerations for S+S and Cloud Computing' in this months Architecture Journal. I co-authored this along with 8 other architects at Microsoft. They were Fred Chong, Alejandro Miguel, Jason Hogg, Ulrich Homann, Brant Zwiefel, Danny Garber, Joshy Joseph and Scott Zimmerman.

Here is the summary: The purpose of this article is to share our thoughts about the design patterns for a new generation of applications that are referred to as Software plus Services, cloud computing, or hybrid computing. The article provides a view into S+S architectural considerations and patterns as they affect common architectural domains such as enterprise, software, and infrastructure architecture.

Check it out at http://msdn.microsoft.com/en-us/architecture/aa699439.aspx

Also be sure to check out the other great articles in the 21st edition of the magazine. This edition is focused on SOA today and tomorrow.

StreamInsight

2009-09-30T07:04:00Z

You are probably asking what is StreamInsight. Well, it is the platform for performing Complex Event Processing (CEP) from Microsoft. Actually to quote exactly what it is - it is a "platform for the continuous and incremental processing of unending sequences of events (event streams) from multiple sources with near-zero latency".

There has been a need to deal with processing at near zero latency for some time. Many times existing products can perform at low latency and that has been acceptable, however, there are a number of situations where low latency wasn't sufficient. That is where Complex Event Processing comes in.

According to Wikipedia CEP is defined as: primarily an event processing concept that deals with the task of processing multiple events with the goal of identifying the meaningful events within the event cloud. CEP employs techniques such as detection of complex patterns of many events, event correlation and abstraction, event hierarchies, and relationships between events such as causality, membership, and timing, and event-driven processes. CEP is to discover information contained in the events happening across all the layers in an organization and then analyze its impact from the macro level as "complex event" and then take subsequent action plan in real time.

Some of the scenarios that call for CEP are:

Manufacturing: Sensors on the plant floor capturing data through device controllers and aggregating data consisting of tens of thousands of events per second with requirements to act on patterns detected to ensure product quality

Web analytics: Instrument server to capture click-stream data and determine online customer behavior

Financial services: Listening to data feeds like news or stocks. Use that data to run queries looking for patterns that find opportunities to buy or sell stock

Some of the features of StreamInsight listed on the download site are:

Derive meaningful and relevant information from data/events streams through complex patterns. These patterns can be defined using a declarative query paradigm based on well-defined streaming semantics with LINQ as query language.
For the development of applications, adapters, and analytics, the user can rely on well-established and powerful development frameworks and tools such as .NET, LINQ, and Microsoft Visual Studio.
The platform integrates with various data sources and sinks through input and output adapters. The framework to build domain-specific adapters utilizes a .NET API to make adoption of the platform easy. Independence between adapters and queries facilitates seamless integration of real-time and historical analysis.
The platform architecture supports a variety of deployment options, from scenarios with a low-footprint embedded option to high-end server deployments.
A rich set of manageability features such as a management interface, a diagnostic interface and a debugging tool are provided as part of the platform.

You can start working with the CTP by downloading it at: http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=a3faa562-b6dc-4702-90c6-bf8e08df3b8b

WCF: Did you know?

2009-08-13T01:15:00Z

Did you know that when doing contract first development and utilize a Boolean data type with a default value (lets just say the default value was set to true), WCF will not serialize out the default value in the message. WCF will only serialize out the value if it differs from the default.

This was an interesting scenario in BizTalk when we were trying to test for the Boolean to be true and we saw that it was actually empty.

Sometimes I wonder with these types of issues if I am the only one in the world that didn't know that. I am going to make the assumption that this will be news to more people then just me.