When retrieving and sending message from or to a Windows Azure
Service Bus Queue, once in a while you might get these two kind of
Queue.Send(message) failed: The token provider was unable to provide a security token while accessing 'https://contoso-sb.accesscontrol.windows.net/WRAPv0.9/' .
Token provider returned message: 'The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.'.
The token provider was unable to provide a security token while accessing 'https://contoso-sb.accesscontrol.windows.net/WRAPv0.9'.
Token provider returned message: 'Error:Code:429:SubCode:T0:Detail:
ACS50000: There was an error issuing a token.
ACS60000: An error occurred while processing rules for relying party 'http://contoso.servicebus.windows.net/bangalore_2_2/' using the service identity or identity provider named 'owner'.
ACS60000: Policy engine execution error.
ACS60021: The request has been terminated by the server (tenant exceeded ratelimit).:TraceID:6be3g937-3203-24gd-691d-26g0e9j7fic5:TimeStamp:2013-07-31 03:17:23Z'.
From the description, it seems you can only acquire token for a set amount of times within a limited time frame. But that's not the case so let me explain what exactly the issue is.
If you notice the errors are being thrown by ACS and if you read my blog you would also have got a pretty good understanding of the same. Service Bus interacts with ACS as documented on this page
So in this case Service Bus is calling ACS and bubbles the error that it receives from ACS. Service Bus in this case acts like a middleware. Service Bus throws these errors to the client application so as not to mask potentially security related exception information from the client application. The client application is required to make use of the retry strategy.
Now coming to the ACS part if you notice the timestamp is for 7/30 . We had an issue in ACS where message “tenant exceeded ratelimit” was returned to clients that did not actually exceed any rate limits. Today, this should not be the case (the bug was fixed), and this error condition will generate correct error code/message as per my blog and the msdn pages mentioned there in. We have also reduced the load on servers by provisioning more instances and the chances of getting this error is very less now. We have now documented the limitations of ACS service at http://msdn.microsoft.com/en-us/library/gg185909.aspx. The page with error codes http://msdn.microsoft.com/en-us/library/gg185949.aspx has also been updated for ACS90046 (ACS busy) or ACS60021.
But the good news is that Service Bus also supports the use of Shared Access Signature(SAS) based authentication. The advantage of using SAS is that unlike ACS they do not have any limits on how many clients/tokens they can have during a cetain time period. The exact details have been very well documented at : Shared Access Signature Authentication with Service Bus and How to use Shared Access Signature Authentication with Service Bus . In case there is no requirement for Federated Authentication then we would actually recommend using the SAS for authentication instead of the default ACS. A complete working sample can be found at http://code.msdn.microsoft.com/windowsazure/Using-Shared-Access-e605b37c