Clemens Vasters, Bldg 42 : Queues

.NET Services March 2009 CTP - Service Bus Routers and Queues - Part 5: The Queue API for the rest of us

clemensv — Tue, 07 Apr 2009 00:38:26 GMT

In Parts 3 and 4 of this series I’ve explained the REST protocol for the .NET Service Bus Queue capability. If that looked a little too complicated for your taste and you’d rather clean with a simple API surface, here’s the API that you’ll prefer.

We’ve got two simple classes in the SDK (in the Microsoft.ServiceBus assembly/namespace) that do the job. The QueueManagementClient allows you to create, renew and delete queues in/from the .NET Service Bus namespace:

  1: public static class QueueManagementClient
  2: {
  3:    public static QueueClient CreateQueue(TransportClientEndpointBehavior credential, 
  4:                                          Uri queueUri, QueuePolicy queuePolicy);
  5:    public static void DeleteQueue(TransportClientEndpointBehavior credential, 
  6:                                   Uri queueUri);
  7:    public static QueueClient GetQueue(TransportClientEndpointBehavior credential, 
  8:                                       Uri queueUri);
  9:    public static QueuePolicy GetQueuePolicy(TransportClientEndpointBehavior credential, 
 10:                                             Uri queueUri);
 11:    public static DateTime RenewQueue(TransportClientEndpointBehavior credential, 
 12:                                      Uri queueUri, TimeSpan requestedExpiration);
 13: }

The most striking difference between this API (and the underlying SOAP protocol for which we still owe you some docs) and the REST API/Protocol is that there’s only a single URI to deal with. The WS-Transfer aligned protocol to manage the policy, the “Queue Transfer” protocol used to dequeue messages, and the “Enqueue” protocol to add messages to the queue are all overlaid over the exact same URI.

The only somewhat ugly element here is the TransportClientEndpointBehavior that’s our awkwardly named credentials container. That class was meant to evolve into something else and then we changed our mind at some point, which is how the baby got stuck with that name. We’ll give that a prettier moniker in one of the next CTPs and as part of an overhaul of the .NET Access Control integration into Service Bus.

The function of the methods themselves should be quite obvious. You can create a queue by applying a policy, you can delete a queue, attach to an existing queue, get the latest queue policy, and renew (extend the lifetime) of a queue. For completeness, here’s CreateQueue in context:

  1: Uri queueUri = ServiceBusEnvironment.CreateServiceUri("sb", solutionName, "/MyQueue/");
  2: 
  3: QueuePolicy queuePolicy = new QueuePolicy();
  4: queuePolicy.ExpirationInstant = DateTime.UtcNow + TimeSpan.FromHours(1);
  5: 
  6: QueueClient client = QueueManagementClient.CreateQueue(credential, queueUri, queuePolicy);
  7: queuePolicy = client.GetPolicy(); // get effective policy

The QueueClient class allows interaction with a Queue. QueueClient instances cannot be created directly, but must be created via the QueueManagementClient factory.

  1: public sealed class QueueClient
  2: {
  3:     public void DeleteLockedMessage(Message message);
  4:     public void DeleteQueue();
  5:     public DateTime GetExpiration();
  6:     public QueuePolicy GetPolicy();
  7:     public Message PeekLock();
  8:     public Message PeekLock(TimeSpan timeout);
  9:     public IEnumerable<Message> PeekLockMultiple(int maxMessages);
 10:     public IEnumerable<Message> PeekLockMultiple(int maxMessages, TimeSpan timeout);
 11:     public void Purge();
 12:     public void ReleaseLock(Message message);
 13:     public DateTime Renew(TimeSpan requestedExpiration);
 14:     public Message Retrieve();
 15:     public Message Retrieve(TimeSpan timeout);
 16:     public IEnumerable<Message> RetrieveMultiple(int maxMessages);
 17:     public IEnumerable<Message> RetrieveMultiple(int maxMessages, TimeSpan timeout);
 18:     public void Send(Message message, TimeSpan timeout);
 19:     public RouterSubscriptionClient SubscribeToRouter(RouterClient routerClient, TimeSpan requestedTimeout);
 20: }

As you can tell, the class is – well – a queue client. It’s using WCF’s Message class as its message abstraction and supports sending messages into the queue, reading messages off the queue in a destructive fashion (Retrieve), and reading messages off the queue using the Peek/Lock pattern which provides resilience against message loss if the message were to be lost in transfer or the receiver fumbled the message. What you’ll also notice is that there are RetrieveMultiple and PeekLockMultiple variants of the retrieval functions which allow for getting more data with fewer network roundtrips.

Again, there should be no surprises using the API. Here’s how you send:

  1: queueClient.Send(Message.CreateMessage(MessageVersion.Default, "Hello", input), TimeSpan.MaxValue);

and here’s how you do a destructive read:

  1: Message message = queueClient.Retrieve();

and here’s how you use the Peek/Lock pattern:

  1: Message message = queueClient.PeekLock();
  2: string content = message.GetBody<string>();
  3: queueClient.DeleteLockedMessage(message);

There are several examples in the SDK showing how to use the API for Queues. The OnewayQueueSender and SoapHttpQueueSender samples are particularly interesting since they just use WCF channels to enqueue, including all the bells and whistles you get from using a WCF channel. Here’s how the OnewayQueueSender does it:

  1: ChannelFactory<IOnewayChannel> channelFactory = 
  2:      new ChannelFactory<IOnewayChannel>(new NetOnewayRelayBinding(), 
  3:                                         new EndpointAddress(queueUri));
  4: channelFactory.Endpoint.Behaviors.Add(userNamePasswordServiceBusCredential);
  5: IOnewayChannel onewayChannel = channelFactory.CreateChannel();
  6: 
  7: string input = Console.ReadLine();
  8: onewayChannel.Hello(new HelloMessage(input));

The logical next question is: So why is the Queue not hooked up to a WCF listener? Answer: That’s what we’ve got “buffered Routers” for. The queue very explicitly provides a ‘pull’ model and the WCF listener would abstract that away and turn it into ‘push’. Routers provide ‘push’ natively and they can embed a Queue. More on Routers in the next set of posts in this series.

.NET Services March 2009 CTP - Service Bus Routers and Queues - Part 4: The REST Queue Protocol in Code Snippets

clemensv — Mon, 06 Apr 2009 23:40:58 GMT

After I’ve gone through the dry facts of the REST Queue Protocol in Part 3 of this series, here’s some code to look at. The code snippets are ripped from the HTTP Queue sample we’ve got in the .NET Services SDK and you can get the copy/paste ready code from there.

The most straightforward way to acquire the required hashed security token (see my comments to Stefan Tilkov) using HttpWebRequest looks like this:

  1: static string HttpGetAuthenticationToken(string username, string password)
  2: {
  3:     string tokenUri = string.Format("https://{0}/issuetoken.aspx?u={1}&p={2}", 
  4:                                     ServiceBusEnvironment.DefaultIdentityHostName, 
  5:                                     username, Uri.EscapeDataString(password));
  6: 
  7:     HttpWebRequest tokenRequest = (HttpWebRequest)WebRequest.Create(tokenUri);
  8:     using (var tokenResponse = tokenRequest.GetResponse())
  9:     {
 10:         using (var tokenResponseStream = tokenResponse.GetResponseStream())
 11:         {
 12:             byte[] tokenBody = new byte[500];
 13:             int tokenBodyLength = tokenResponseStream.Read(tokenBody, 0, 500);
 14:             return Encoding.UTF8.GetString(tokenBody, 0, tokenBodyLength);
 15:         }
 16:     }
 17: }
 18:

Once you’ve got the token, you need to pick a place in the Service Bus namespace. We’ve got a built-in helper function in the SDK that sits on ServiceBusEnvironment and which knows all the right incantations:

  1:  Uri queueUri = ServiceBusEnvironment.CreateServiceUri(
  2:                    Uri.UriSchemeHttps, 
  3:                    solutionName, 
  4:                    "/MyHttpQueue/");

Invoked in this way, the method yields the URI https://yourproject.servicebus.windows.net/MyHttpQueue/ so there’s spectacularly little magic to it. The reason why we recommend that you use the method in .NET applications is that we’ve broken everyone’s apps going from the previous CTP to the current CTP due to the namespace restructuring and we’d like to avoid doing that again as we improve the namespace story. I don’t foresee any further change of the magnitude we had this time, however.

To create a new Queue, you first create a Queue policy and then apply it to the Service Bus namespace name. We’ve got a Queue Policy object in the SDK that I’ll use here to keep things short. I’m accepting all the defaults that are spelled out in Part 2 of the series, but explicitly I’m setting the expiration to 1 hour. The Queue Policy XSD schema in the the HTTP Queue sample’s readme document, by the way.

  1: QueuePolicy queuePolicy = new QueuePolicy();
  2: queuePolicy.ExpirationInstant = DateTime.UtcNow + TimeSpan.FromHours(1);
  3: queueManageUri = HttpCreateQueue(token, queueUri, queuePolicy);

The code for creating the Queue shouldn’t be very surprising. It’s a POST of an Atom 1.0 entry for which I’m using HttpWebRequest and the WCF 3.5 Syndication API. The SDK sample is using a rather terse version of the ‘correct’ way of doing things. I’m quoting the expanded variant from the Text140 sample that I showed at MIX’09:

  1: public static QueuePolicy HttpCreateQueue(string token, Uri queueUri, QueuePolicy policy, 
  2:                                           out Uri queueSelfUri, out Uri queueHeadUri)
  3: {
  4:     HttpWebRequest webRequest;
  5:     HttpWebResponse webResponse;
  6: 
  7:     queueSelfUri = null;
  8:     queueHeadUri = null;
  9: 
 10:     // Create a new syndication item and add the queue policy as an extension
 11:     SyndicationItem syndicationItem = new SyndicationItem();
 12:     syndicationItem.ElementExtensions.Add(
 13:         new SyndicationElementExtension("QueuePolicy",
 14:                                         "http://schemas.microsoft.com/ws/2007/08/connect",
 15:                                         policy));
 16: 
 17:     // create a new POST request and set the Content-Type to represent an Atom 1.0 entry
 18:     webRequest = HttpWebRequest.Create(queueUri) as HttpWebRequest;
 19:     webRequest.ConnectionGroupName = "queueclient";
 20:     webRequest.KeepAlive = false;
 21:     webRequest.Method = "POST";
 22:     webRequest.Headers.Add("X-MS-Identity-Token", token);
 23:     webRequest.ContentType = "application/atom+xml;type=entry;charset=utf-8";
 24: 
 25:     // write the item to the request stream
 26:     using (var requestStream = webRequest.GetRequestStream())
 27:     {
 28:         using (var textWriter = new XmlTextWriter(requestStream, Encoding.UTF8))
 29:         {
 30:             syndicationItem.GetAtom10Formatter().WriteTo(textWriter);
 31:             textWriter.Flush();
 32:         }
 33:     }
 34: 
 35:     // get the response
 36:     using (webResponse = webRequest.GetResponse() as HttpWebResponse)
 37:     {
 38:         return GetQueueInfoFromResponse(webResponse, queueUri, out queueSelfUri, out queueHeadUri);
 39:     }
 40: }
 41: 
 42: static QueuePolicy GetQueueInfoFromResponse(HttpWebResponse webResponse, Uri queueUri, 
 43:                                                     out Uri queueSelfUri, out Uri queueHeadUri)
 44: {
 45:     queueHeadUri = null;
 46:     queueSelfUri = null;
 47: 
 48:     if (webResponse.ContentType.StartsWith("application/atom+xml;type=entry", StringComparison.OrdinalIgnoreCase))
 49:     {
 50:         Atom10ItemFormatter atomItemFormatter = new Atom10ItemFormatter();
 51:         using (var responseReader = new XmlTextReader(webResponse.GetResponseStream()))
 52:         {
 53:             atomItemFormatter.ReadFrom(responseReader);
 54:             // we found it. let's get the policy and the queue head URI
 55:             foreach (SyndicationLink queueSelfLink in
 56:                            from link in atomItemFormatter.Item.Links
 57:                            where link.RelationshipType.Equals("self", StringComparison.OrdinalIgnoreCase)
 58:                            select link)
 59:             {
 60:                 queueSelfUri = queueSelfLink.Uri;
 61:                 break;
 62:             }
 63: 
 64:             foreach (SyndicationLink queueHeadLink in
 65:                            from link in atomItemFormatter.Item.Links
 66:                            where link.RelationshipType.Equals("queuehead", StringComparison.OrdinalIgnoreCase)
 67:                            select link)
 68:             {
 69:                 queueHeadUri = queueHeadLink.Uri;
 70:                 break;
 71:             }
 72: 
 73:             return atomItemFormatter.Item.ElementExtensions.ReadElementExtensions<QueuePolicy>(
 74:                  "QueuePolicy",
 75:                  "http://schemas.microsoft.com/ws/2007/08/connect").FirstOrDefault();
 76:         }
 77:     }
 78:     else
 79:     {
 80:         Atom10FeedFormatter atomFeedFormatter = new Atom10FeedFormatter();
 81:         using (var responseReader = new XmlTextReader(webResponse.GetResponseStream()))
 82:         {
 83:             atomFeedFormatter.ReadFrom(responseReader);
 84: 
 85:             // look for an item whose alternate-link equals the desired queue URI. The alternate 
 86:             // URI is the 'tail' endpoint by which the queue is mapped into the namespace. We are 
 87:             // comparing Authority and Path since the scheme might be http or https and we want to match
 88:             // either.
 89:             foreach (SyndicationItem requestedItem in
 90:                          from item in atomFeedFormatter.Feed.Items
 91:                          from link in item.Links
 92:                          where 
 93:                            (link.RelationshipType.Equals("alternate", StringComparison.OrdinalIgnoreCase) &&
 94:                             link.Uri.Authority.Equals(queueUri.Authority, StringComparison.OrdinalIgnoreCase) &&
 95:                             link.Uri.AbsolutePath.Equals(queueUri.AbsolutePath, StringComparison.OrdinalIgnoreCase)) ||
 96:                            (link.RelationshipType.Equals("self", StringComparison.OrdinalIgnoreCase) && 
 97:                             link.Uri.Equals(queueUri))
 98:                          select item)
 99:             {
100:                 // we found it. let's get the policy and the queue head URI
101:                 foreach (SyndicationLink queueSelfLink in
102:                                from link in requestedItem.Links
103:                                where link.RelationshipType.Equals("self", StringComparison.OrdinalIgnoreCase)
104:                                select link)
105:                 {
106:                     queueSelfUri = queueSelfLink.Uri;
107:                     break;
108:                 }
109: 
110:                 foreach (SyndicationLink queueHeadLink in
111:                                from link in requestedItem.Links
112:                                where link.RelationshipType.Equals("queuehead", StringComparison.OrdinalIgnoreCase)
113:                                select link)
114:                 {
115:                     queueHeadUri = queueHeadLink.Uri;
116:                     break;
117:                 }
118: 
119:                 return requestedItem.ElementExtensions.ReadElementExtensions<QueuePolicy>(
120:                      "QueuePolicy",
121:                      "http://schemas.microsoft.com/ws/2007/08/connect").FirstOrDefault();
122:             }
123:         }
124:     }
125:     return null;
126: }

The “magic” that isn’t straight up Atom Pub is in lines 10-15 above. I’m attaching the policy object to the SyndicationItem abstraction that will be written out as an Atom 1.0 entry. [I know that copy/paste is difficult using this format – I need to clean up that expanded utility class and will post it in downloadable form within the next few days]

The method returns the effective policy that the Service Bus is using, and yields the management URI for the queue where the Atom entry resides (self-link) as well as the head of the queue as output arguments. The queueUri to which you applied the entry has morphed into the Queue’s tail when the method returns successfully. Having the Queue by its tail, submitting messages into it is very simple. Just do a plain POST. Here we’re just writing a simple string from the local variable input as the entity body and send it.

  1: // send
  2: HttpWebRequest sendRequest = HttpWebRequest.Create(queueUri) as HttpWebRequest;
  3: sendRequest.Method = "POST";
  4: sendRequest.Headers.Add("X-MS-Identity-Token", token);
  5: sendRequest.ContentType = "text/plain;charset=utf-8";
  6: using (var sendStream = sendRequest.GetRequestStream())
  7: {
  8:     using (var writer = new StreamWriter(sendStream, Encoding.UTF8))
  9:     {
 10:         writer.Write(input);
 11:         writer.Flush();
 12:     }
 13: }
 14: sendRequest.GetResponse().Close();

For a simple destructive read, you use the ‘queue head’ URI and do a DELETE:

  1: HttpWebRequest dequeueRequest = 
  2:     HttpWebRequest.Create(queueHeadUri.AbsoluteUri+
  3:                           "?encoding=asreply&maxmessages=1&timeout=60") as HttpWebRequest;
  4: dequeueRequest.ConnectionGroupName = "dequeue";
  5: dequeueRequest.Method = "DELETE";
  6: dequeueRequest.ContentLength = 0;
  7: dequeueRequest.Headers.Add("X-MS-Identity-Token", token);
  8: using (HttpWebResponse response = dequeueRequest.GetResponse() as HttpWebResponse)
  9: {
 10:     if (response.StatusCode == HttpStatusCode.OK)
 11:     {
 12:         using (var responseBody = response.GetResponseStream())
 13:         {
 14:             using (var responseReader = new StreamReader(responseBody, Encoding.UTF8))
 15:             {
 16:                 string data = responseReader.ReadToEnd();
 17:                 Console.WriteLine(data);
 18:             }
 19:         }
 20:     }
 21: }

For a peek/lock read where you first lock the message and then delete the lock when you want to keep the message, you do a POST on the head and remember the X-MS-Message-Lock header value:

  1: HttpWebRequest dequeueRequest = 
  2:      HttpWebRequest.Create(queueHeadUri.AbsoluteUri+
  3:                            "?encoding=asreply&maxmessages=1&timeout=60") as HttpWebRequest;
  4: dequeueRequest.ConnectionGroupName = "dequeue";
  5: dequeueRequest.Method = "POST";
  6: dequeueRequest.ContentLength = 0;
  7: dequeueRequest.Headers.Add("X-MS-Identity-Token", token);
  8: using (HttpWebResponse response = dequeueRequest.GetResponse() as HttpWebResponse)
  9: {
 10:     if (response.StatusCode == HttpStatusCode.OK)
 11:     {
 12:         lockUri = new Uri(response.Headers["X-MS-Message-Lock"]);
 13:         using (var responseBody = response.GetResponseStream())
 14:         {
 15:             using (var responseReader = new StreamReader(responseBody, Encoding.UTF8))
 16:             {
 17:                 string data = responseReader.ReadToEnd();
 18:                 Console.WriteLine(data);
 19:             }
 20:         }
 21:     }
 22: }

If you decide to keep the message (i.e. your app didn’t puke processing it), you DELETE the lock, otherwise you do a PUT. The code is practically the same except for the method so I’ll just quote DELETE:

  1: HttpWebRequest deleteLockedMessageRequest = HttpWebRequest.Create(lockUri) as HttpWebRequest;
  2: deleteLockedMessageRequest.ConnectionGroupName = "lock";
  3: deleteLockedMessageRequest.Method = "DELETE";
  4: deleteLockedMessageRequest.KeepAlive=false;
  5: deleteLockedMessageRequest.ContentLength = 0;
  6: deleteLockedMessageRequest.Headers.Add("X-MS-Identity-Token", token);
  7: deleteLockedMessageRequest.ContentLength = 0;
  8: deleteLockedMessageRequest.GetResponse().Close();

That’s it. Create, Enqueue, Destructive Read, Peek/Lock Read, and .. oh .. yes.. forgot Delete the Queue. Take the snippet above and swap out the lockUri for the queueSelfUri. Done.

Ah, yes, and please … don’t blame me for HTTP or HttpWebRequest or Clemens’ demo coding style requiring too many lines of code. I could have applied a bit of code-compression here, but I’m intentionally trying not to abstract away too much of the protocol here. I’m generally with you, though. The guys putting together the WCF REST Starter Kit are working on making code like that shrink.

.NET Services March 2009 CTP - Service Bus Routers And Queues - Part 3: The REST Queue Protocol

clemensv — Thu, 02 Apr 2009 16:08:13 GMT

[If this text looks vaguely familiar you’ve read the HttpQueueSample Readme doc from the SDK. Good job!]

Here is the Service Bus Queue REST protocol. I apologize if this is a bit dry, but I want to give you the bare protocol facts first and will follow-up with code samples later. Look at the HTTP Queue SDK sample if you can’t wait for code ;-)

You can find the HTTP Queue sample in the SDK install path under .\Samples\ServiceBus\ExploringFeatures\Queues\HttpMessages\

The beauty of REST is that there’s a uniform interface. It’s all just GET, POST, PUT, DELETE, a set of resources, and a rigorous application of the RFC2616 / HTTP 1.1 rules. That doesn’t make the description particularly entertaining, but it does yield a pretty consistent interaction model:

Authorization

All Queue operations require the client to be appropriately authorized. The client must therefore acquire an lightweight identity token from the Microsoft .NET Access Control Service using the .NET Services solution name and the solution password and include the acquired token with each request. The tokens are added to requests using the X-MS-Identity-Token HTTP header. The value for this header is a short-lived token which can be used for a period of up to 8 hours and must be acquired from the .NET Access Control Service. We strongly encourage you to guard the acquired tokens in the same fashion as you would guard a credential and you should not openly expose them unprotected on the network or web-pages or embedded in a Flash or Silverlight application.

The request for acquiring the token is an HTTPS GET on the URI https://accesscontrol.windows.net/issuetoken.aspx?u={solutionName}&p={solutionPassword} whereby you replace the arguments for solution name and the solution password with your .NET Services solution credentials. If a token is successfully issued to you the request returns with a 200 (OK) status code and contains a text/plain entity body with a short, base64 encoded token hash as a single line of text, which you for the value of the X-MS-Identity-Token header.

Creating a Queue

A Queue is created on the Service Bus in four simple steps. I’ve discussed most of this in the post on Queue Policies:

Select a name in the Service Bus naming hierarchy where the Queue should be located, i.e. https://mysolution.servicebus.windows.net/myapp/q1
Create a Queue policy and define the desired properties of the Queue.
Embed the Queue policy into an ATOM 1.0 <atom:entry> as an extension and POST the entry to the designated Queue URI with the content-type application/atom+xml;type=entry. The request must carry an X-MS-Identity-Token header and the respective identity must have 'Manage' permission on for scope covering the Queue URI.
If the queue was successfully created, the POST request returns with a 201 'created' status code along with a Location header. The response entity body contains the effective Queue policy as applied to the name by the Service Bus.

The location header contains the queue's management URI that you need to retain in your application state to have access to the queue's metadata and management functions. If the queue could not be created successfully, the request may yield the following status codes:
- 400 - Bad Request. The policy was malformed or invalid.
- 403 - Forbidden. The client did not provide a X-MS-Identity-Token header, the provided token is no longer valid for use, or the provided identity is not authorized to create a Queue at this location.
- 409 - Conflict. There is already a Queue with an incompatible policy at the given location or the location is occupied by a Router or a Service Bus listener.
- 415 - Unsupported Media Type. The request did not carry the required content-type header.
- 500 - Internal Server Error. The processing failed to to a condition internal to the Service Bus service.

REST Anatomy of a Queue

Each Queue has five distinct types of resources: Policy, Tail, Head, Locks, and Control. The concrete URIs are discoverable using the Service Registry's Atom Publishing protocol. you should not make any assumptions about the particular format of the URIs spelled out below since the particular format might change.

The Atom 1.0 entry shown below is also enclosed in the response entity body of the create request (POST) explained above. A Queue's representation in the registry feed as seen when doing a GET (discovery-) request on the queue's parent URI is commonly as follows:

<feed xmlns="http://www.w3.org/2005/Atom">
  ...
  <entry>
    <id>{id}</id>
    <title type="text">MyHttpQueue</title>
    <updated>{date}</updated>
    <link rel="alternate" href="https://solution.servicebus.windows.net/myapp/q1" />
    <link rel="self" href="https://solution.servicebus.windows.net/myapp/q1!(queue)" />
    <link rel="queuehead" href="https://solution.servicebus.windows.net/myapp/q1!(queue/head)" />
    <link rel="queuecontrol" href="https://solution.servicebus.windows.net/myapp/q1!(queue/control)" />
    <QueuePolicy xmlns="http://schemas.microsoft.com/ws/2007/08/connect">
      <Discoverability>Public</Discoverability>
      <ExpirationInstant>{expiration}</ExpirationInstant>
    </QueuePolicy>
  </entry>
</feed>

Managing The Queue

<link rel="self" href="https://solution.servicebus.windows.net/myapp/q1!(queue)" />

The "self" link in the entry above is the queue's management URI that allows you to interact with the queue's policy.

To renew a queue and extend its expiration you PUT an updated Atom 1.0 entry with the effective policy and an updated ExpirationInstant value to the "self" URI. The "self" URI is the same URI as the one returned in the Location header returned by the create POST explained above. The request will yield a 200 status code if the renewal is successful. It will yield a 404 if the queue no longer exists.
To delete a queue you issue a DELETE request on the queue's management URI. The Delete request is sent without any entity body and MUST have a Content-Length header that is set to zero (0). Not setting this header will yield a 411 status code. The request yields a 204 response if the queue was deleted successfully.

Enqueue

<link rel="alternate" href="https://solution.servicebus.windows.net/myapp/q1" />

The "alternate" link refers to the tail of the queue. The Queue's tail resource is where senders submit messages into the Queue. The queue's tail URI is identical to the URI where the queue was initially created. The queue's tail endpoint accepts requests with any HTTP Content-Type and any HTTP method except GET, HEAD, and OPTIONS.

The endpoint should be considered a 'delegate' of the message recipient and not the resource that the respective operations have an immediate effect on. That means that the HTTP method applied to the Queue tail doesn’t have any particular semantic. If a message has been successfully accepted into the queue, the request returns a 202 status code. Otherwise it will return a 500

Dequeue and Peek/Lock

<link rel="queuehead" href="https://solution.servicebus.windows.net/myapp/q1!(queue/head)" />

The "queuehead" link refers to the head of the Queue. The Queue's head resource is where receivers retrieve messages from the queue. It is modeled as a resource whose current representation corresponds to the message(s) that reside at the head of the queue and which is/are next in line to be retrieved by consumers. Permitted operations on the head are:

DELETE: Delete performs a destructive read on the queue whereby the message(s) at the head of the queue are permanently deleted and returned as the entity body of the response to the request. The content type and formatting of the response depends on a set of query parameters that are discussed below. The DELETE request's response status code is 200 if at least one message could be retrieved and deleted from the queue and the message content is enclosed in the response's entity body. The response status code is 204 if no message could be retrieved/deleted.
POST: Post creates a lock on the message(s) at the head of the queue and returns the locked messages. Locked messages are temporarily removed from the head of the queue for a period of 1 minute during which time the receiver can decide whether to return the message to the head of the queue or whether to permanently delete it. The POST request's response status code is 200 if at least one message could be retrieved and locked on the queue and the message content is enclosed in the response's entity body. The response status code is 204 if no message could be retrieved/locked.

Releasing and Deleting Peek/Lock Message Locks

Any retrieved, peek/locked message (POST on “queuehead”) contains an HTTP header X-MS-Message-Lock whose value is a URI. The following operations can be performed on this URI:

DELETE: The message has been accepted/processed by the receiver and the lock shall be deleted. The deletion of the lock discards the message permanently from the queue. The Delete request is sent without any entity body and MUST have a Content-Length header that is set to zero (0). Not setting this header will yield a 411 status code. The request yields a 204 response if the lock was deleted successfully.
PUT: The message has been not been accepted/processed by the receiver and the message shall be put back at the head of the queue. This operation transitions the lock from the locked state into the unlocked state which releases the message back into the queue. The lock itself is discarded afterwards. The Put request is sent without any entity body and MUST have a Content-Length header that is set to zero (0). Not setting this header will yield a 411 status code. The request yields a 204 response if the lock was released successfully.

Options for Dequeue and Peek/Lock

The DELETE and POST operation have a set of options that are expressed as query parameters appended to the queue's head URI. The options are the same for both operations:

timeout=value - The value is a numerical value (expressed in seconds) that indicates how long the client is willing to wait for the polling request to complete. The value should not exceed 60 seconds and a value of 30 seconds is a safer choice with intermediate proxies in place. If the timeout expires and no message is available, the request completes with a 204 response code. The default value is zero (0), which means that the request returns immediately.
maxmessages=value - Indicates how many messages the client is willing to accept in a single response. The default is 1. The service will return at most 10 messages at a time and a value greater than zero requires encoding=multipart (see below)
encoding=value - There are three supported encoding modes for the REST retrieval model
- asreply - The request stored in the Queue is mapped onto the response to the DELETE/POST request returning the message from the queue. This option requires maxmessages=1 or omission of the maxmessages option.
- single - The request stored in the queue is returned as a complete HTTP request frame on the response to the DELETE/POST request using Content-Type application/http. This option requires maxmessages=1 or omission of the maxmessages option.
- multipart - The request(s) stored in the queue are returned as a complete HTTP request frames on the response to the DELETE/POST request using Content-Type multipart/mixed with MIME parts of Content-Type application/http.

Queue Content Control

<link rel="queuecontrol" href="https://solution.servicebus.windows.net/myapp/q1!(queue/control)" />

The "queuecontrol" link refers to the control resource provides information about the status of the queue and allows for purging all of the queue's contents. The control resource is present in the current CTP, but you cannot perform any operations on it.

.NET Services March 2009 CTP - Service Bus Routers And Queues - Part 2: Queue Policies

clemensv — Thu, 02 Apr 2009 14:10:25 GMT

In the previous post in this series I’ve discussed some of the foundational principles of the new Queue and Router features of the .NET Services Bus and the role that policies play in turning names in the namespace into messaging primitives. In this post I’ll start drilling into the capabilities of the Queue feature and will discuss what’s in the queue policy.

The capabilities of the Queue are best explored by looking at the Queue policy. Policies can be expressed in two ways: You can either you use the .NET Services SDK’s Microsoft.ServiceBus.QueuePolicy class with a .NET application or you can just whip up a bit of XML. The structure and the serialized representation of that class is exactly equivalent to an XML queue policy you’d wire up “by hand” and thus I’ll discuss the policy in terms of its XML elements:

The queue policy has the element name “QueuePolicy” within the XML namespace “http://schemas.microsoft.com/ws/2007/08/connect”. That’s the namespace we use for most protocols and data structures in the .NET Service Bus at this development stage and you should expect that we’re migrating to the “final” V1 namespaces in one of the next CTPs. If you use the .NET class you’ll just have to recompile to snap to new namespaces once they come around.

Below is a list of the policy’s elements and the permitted values. All elements are optional and can appear at most once. The default value applies when an element is omitted. It’s perfectly ok to send an empty QueuePolicy and accept the default values to keep things simple. I’d recommend to always set the ExpirationInstant value, however.

One required forward reference: Don’t get confused or distracted trying to figure out from here what it means “to provide an authorization token”. The docs explain this and I’ll also address this in the next blog post when I dive into the protocol.

Authorization – The authorization setting indicates whether sender and/or consumers must provide a security token when interacting with the queue. You can explicitly permit the .NET Service Bus to allow anonymous consumers and/or anonymous senders on a Queue. If a token is required, a sender must present a token with “Send” permission that is issued by the .NET Access Control service and the consumer must present a token with “Listen” permission that is issued by the .NET Access Control service. The respective rules are set up in the .NET Access Control service on your project’s “Service Bus” scope. Mind that all anonymous traffic is attributed to the “manager” of a Queue, i.e. to whoever sets the policy; all traffic is metered and accounted for. The permitted values for this element are:
- Required – (default) Senders and Consumers must provide an authorization token to send/retrieve messages.
- RequiredToSend – Senders must provide an authorization token with a “Send” claim to send messages, consumers don’t.
- RequiredToReceive – Consumers must provide an authorization token with a “Listen” claim to receiver messages, senders don’t.
- NotRequired – Neither senders nor consumers must provide a token, i.e. the queue is set up for anonymous traffic.
Discoverability – The discoverability setting defines under which conditions the Queue and its policy are visible in the Service Registry Atom feed and any forthcoming discoverability mechanisms. When you are browsing the Service Registry feed from within a browser, the only setting that will make the Queue visible is Public since the browser isn’t able to attach an authorization token without some scripting assistance.
- Managers - (default) The Queue and its policy are only discoverable if you provide an authorization token with the request and the token carries a “Manage” claim.
- ManagersListeners – The Queue and its policy are only discoverable if you provide and authorization token with the request and the token carries a “Manage” and/or “Listen” claim.
- ManagersListenersSenders – The Queue and its policy are only discoverable if you provide and authorization token with the request and the token carries a “Manage” and/or “Listen” and/or “Send” claim.
- Public – The Queue and its policy are discoverable without any authorization requirement, i.e. the general public.
ExpirationInstant – This is an XML dateTime value indicating the TTL (time-to-live) for the Queue. You can make Queues long-lived or short-lived. A Queue’s lifetime may be extended by changing this policy value and reapplying the policy. Once a Queue expires it is removed from the system along with all the messages that reside in it. The system may limit the Queue lifetime in the effective policy that you get back from the create/update request, but we’ll typically allow at least 24 hours without renewal. This value has a default of 24 hours, a maximum of 21 days and a minimum of 30 seconds. The value must represent a future dateTime and must be expressed in UTC.
MaxMessageSize – This is a numeric value defining the maximum size of any individual message that can be accepted into the Queue. The maximum message size is expressed in bytes and includes all infrastructure-imposed and encoding-related overhead. The actual overhead varies significantly based on whether messages are sent as SOAP 1.1, SOAP 1.2, or plain HTTP frames using text or binary encoding and whether a security token is required. A very defensive assumption is to reserve 10-12KB for protocol overhead in complex cases with WS* Security; the minimal allocation for protocol overhead should be around 4KB. The default and maximum values for the maximum message size is 60KB (60*1024 bytes). The minimum value is 8KB. We suggest that the payload size for an individual message does not exceed 48KB, even though you can try to push it a bit. The expectation is that these values will trend up but I don’t see them more than doubling due to throughput, timeliness and scale considerations. If your data is larger you should consider how you can chunk it up.
TransportProtection – The transport protection setting defines whether senders and consumers must use HTTPS to interact with the Queue. This is the default setting. You would modify this to accommodate clients – typically on devices - that don’t do HTTPS all that well. Permitted values:
- AllPaths - (default) Any interaction with the queue requires HTTPS.
- None – Any interaction may be performed using either HTTP or HTTPS.
EnqueueTimeout – The enqueue timeout is an XML duration value that specifies how long an enqueue operation will hang if the queue is at capacity (full). The default value is 10 seconds, the minimum is 0 seconds, and the maximum is 60 seconds. If the timeout expires and the message could not be added to the queue during that time, the queue will act according to the Overflow policy setting.
MaxConcurrentReaders - [this setting not supported/enforced in the March 2009 CTP] The maximum concurrent readers value defines how many concurrent readers are permitted on the queue. If this numeric value is smaller than the default value of 2^31 (max int), the queue protocol will switch into ‘session mode’ that grants a limited number of read-locks on the queue. The minimum value is 1.
MaxDequeueRetries - [this setting not supported/enforced in the March 2009 CTP] The maximum dequeue retries value defines how often a message may be peek/locked and put back into the queue until it is considered poisonous, i.e. after how many retries it should be expected the the consumer will not be able to ever consume the message successfully, because it is malformed or the consumer experiences an error condition that requires some form of manual intervention (including a bug fix). If the message is found to be poisonous it will be sent once and without any retries to the endpoint specified in the policy’s PoisonMessageDrop value.
MaxMessageAge – The maximum message age is an XML duration value indicating after what time any enqueued message is considered ‘stale” and will be automatically dropped and removed from the queue. The default value is 10 minutes (600 seconds), the minimum value is 0 seconds (which effectively means that all incoming messages get dropped), the maximum value is 7 days.
MaxQueueCapacity – This numeric value indicates the maximum size of the Queue in bytes. This is a system-calculated value that cannot be set on the .NET class and should not be set by clients creating policies from scratch. The default and maximum capacity of any queue is currently capped at 2MB. This is a limitation specific to the March 2009 CTP , it’s been painful to impose this constraint, and it’s absolutely expected that this limit will be expanded significantly. 2MB still allow for a several hundred notification messages of a 2-4KB. If you need to store more data you can absolutely create several queues and partition data across them either explicitly or using Routers with a load-balancing message distribution policy (more on that in a subsequent post).
MaxQueueLength – This numeric value indicates the maximum queue length. The maximum and default value is 2^31, the minimum value is 1. We’re not enforcing the exact queue length in the March 2009 CTP, but your code should assume that we do. The queue length value is the basis for the calculation of the MaxQueueCapacity, with MaxQueueCapacity = min(MaxQueueLength * MaxMessageSize, 2MB) enforcing the hard 2MB limit that is currently in effect. You don’t need to touch this value unless you’d really want a Queue that’s even more size-constrained.
Overflow – The overflow policy setting becomes relevant when the queue is at capacity and the EnqueueTimeout has expired. It tells the Queue what to do with the message that just came in and that can’t be put into the queue because it’s full. Permitted values:
- RejectIncomingMessage - (default) The message will be rejected and an error status code or SOAP fault will be sent to the sender.
- DiscardIncomingMessage – The sender will get an indication that the message has been accepted, but the message will be dropped on the floor.
- DiscardExistingMessage – The Queue will remove and discard messages from the head of the Queue until the new, incoming message fits in the Queue.
PoisonMessageDrop – [this setting not supported in the March 2009 CTP] This value is a WS-Addressing 1.0 EndpointReference referring to an endpoint that any poison messages will be sent to once the MaxDequeueRetries limit has been exceeded. The EndpointReference may point to any SOAP or plain HTTP endpoint that can accept ‘any’ message.

Phew! Lots of options. The good thing is that most apps should be ok with the defaults.

I know that the 2MB capacity limit is somewhat disappointing and I’m certainly not happy with it. There’s a particular behavior (with bug number) that may occur under very rare circumstances in the underlying replication system, which caused us to play it very safe instead of risking data loss. I don’t think the limit is a showstopper for apps that send notifications and events around – it is a showstopper for apps that want to exchange larger payloads and we’re working to relax that limit and make Queues much, much larger as soon as we can. You can obviously always spool larger data into SDS or one of the Azure storage systems and then send a reference to that data as a message, but it’d be strange for a messaging system to make that a required pattern for data of all sizes. If we’re talking hundreds of megabytes it makes sense, though.

With the due apology out of the way, let’s look at how a policy may be applied to a namespace name – or in other words, how a queue is created in the simplest case (this must be done via HTTPS). The model here is that the client proposes a policy and Service Bus is at liberty to adjust the policy.

If you are intimately familiar with Atom, you’ll notice that the <QueuePolicy> is an extension and isn’t carried as <content>. That’s by intent. <content> is for people, extensions are for apps. We’ll start using <content> in a later milestone, so consider that being reserved.

HTTP/1.1 POST /myapp/q1
Host: clemensv.servicebus.windows.net
X-MS-Identity-Token: A6hbJklu18hsnHRql61k1==
Content-Type: application/atom+xml;type=entry;charset=utf-8
Content-Length: nnn

<entry xmlns=”http://www.w3.org/2005/Atom”>
   <QueuePolicy xmlns=”http://schemas.microsoft.com/ws/2007/08/connect”>
      <ExpirationInstant>2009-04-03T12:00:00</ExpirationInstant>
   </QueuePolicy>
</entry>

If the queue can be created, the response is a 200 and you get the policy back along with any adjustments that the service may make. This is called the “effective” policy, i.e. that’s what the server is using. You also learn about where you can modify the policy, since – as I explained before – the endpoint where you POST the policy to is morphing into the Queue’s tail.

HTTP/1.1 200 OK
Location: https://project-name.servicebus.windows.net/myapp/q1!(queue)
Content-Type: application/atom+xml;type=entry;charset=utf-8
Content-Length: nnn

<entry xmlns=”http://www.w3.org/2005/Atom”>
   <link rel=”self” href=”https://project-name.servicebus.windows.net/myapp/q1!(queue)” />
   <link rel=”alternate” href=”https://project-name.servicebus.windows.net/myapp/q1” />
   <link rel=”queuehead” href=”https://project-name.servicebus.windows.net/myapp/q1!(queue/head)” />
   <QueuePolicy xmlns=”http://schemas.microsoft.com/ws/2007/08/connect” >
         <ExpirationInstant>2009-04-03T12:00:00</ExpirationInstant>
         <MaxQueueCapacity>2097152</MaxQueueCapacity>
   </QueuePolicy>
</entry>

If you want to renew the Queue (extend the expiration), take the effective policy, adjust ExpirationInstant and do a PUT to the “self” location.

HTTP/1.1 PUT /myapp/q1!(queue)
Host: project-name.servicebus.windows.net
X-MS-Identity-Token: A6hbJklu18hsnHRql61k1==
Content-Type: application/atom+xml;type=entry;charset=utf-8
Content-Length: nnn

<entry xmlns=”http://www.w3.org/2005/Atom”>
   <link rel=”self” href=”https://project-name.servicebus.windows.net/myapp/q1!(queue)” />
   <link rel=”alternate” href=”https://project-name.servicebus.windows.net/myapp/q1” />
   <link rel=”queuehead” href=”https://project-name.servicebus.windows.net/myapp/q1!(queue/head)” />
   <QueuePolicy xmlns=”http://schemas.microsoft.com/ws/2007/08/connect” >
         <ExpirationInstant>2009-04-04T12:00:00</ExpirationInstant>
         <MaxQueueCapacity>2097152</MaxQueueCapacity>
   </QueuePolicy>
</entry>

[Bug note: The PUT will return the new effective policy just like the POST response shown above, but the returned Atom <links> aren’t correctly formed. Keep the ones returned from the POST]

If you want to delete the queue, just nuke the policy:

HTTP/1.1 DELETE /myapp/q1!(queue)
Host: project-name.servicebus.windows.net
X-MS-Identity-Token: A6hbJklu18hsnHRql61k1==
Content-Length: 0

So that’s the policy story for Queues. In the next posts I’ll discuss the REST Queue protocol and the SOAP Queue protocol for how you send message to the queue and get messages out. REST I’ll explain in protocol terms, the SOAP model in .NET programming model terms.