Applications using Azure cache might commonly encounter exceptions similar to below while performing cache operations like get, put. We refer to these kind of errors as transient errors.  Developer is required to implement retry logic in the application code to shield the application from transient errors.

ErrorCode<ERRCA0017>:SubStatus<ES0006>:There is a temporary failure. Please retry later. (One or more specified cache servers are unavailable, which could be caused by busy network or servers. For on-premises cache clusters, also verify the following conditions. Ensure that security permission has been granted for this client account, and check that the AppFabric Caching Service is allowed through the firewall on all cache hosts. Also the MaxBufferSize on the server must be greater than or equal to the serialized object size sent from the client.)

Thanks to "Transient fault handling application block" from patterns & practices team. It is now very easy to implement sophisticated retry logic in your application. I'm describing the instructions in 3 steps here.

  1. Add the Transient Fault Handling Application Block to your application
    1. Install latest version of NuGet from here http://nuget.org/.
    2. Add a reference to the Transient Fault Handling Application Block assemblies. In Microsoft Visual Studio, right-click your project node in Solution Explorer, and then click Manage NuGet Packages.
    3. Click the Online button, and then in the Search Online box, type topaz.
    4. Click the Install button for the Enterprise Library 5.0 - Transient Fault Handling Application Block package.
    5. Read and accept the license terms for the packages listed.
    6. After NuGet has finished installing the packages, click Close.
    7. NuGet has now updated your project with all the necessary assemblies and references that you need to use the Transient Fault Handling Application Block. The project now also includes a readme file that contains important information about the Transient Fault Handling Application Block.
  2. Configure retry policy for your application.
    //Add below using statements to your project/module.

    using Microsoft.Practices.TransientFaultHandling;
    using Microsoft.Practices.EnterpriseLibrary.WindowsAzure.TransientFaultHandling;
    using Microsoft.Practices.EnterpriseLibrary.WindowsAzure.TransientFaultHandling.Cache;

     // I've defined retryStrategy, retryPolicy in page_load event. It is really up to your application requirement where you would like to define these

     // Define your retry strategy: in this sample, I'm retrying operation 3 times with 1 second interval
     var retryStrategy = new FixedInterval(3, TimeSpan.FromSeconds(1));

     // Define your retry policy here. This sample uses CacheTransientErrorDetectionStrategy
     var retryPolicy = new RetryPolicy<CacheTransientErrorDetectionStrategy>(retryStrategy);

     

  3. Make azure cache calls inside ExecuteAction delegate as shown below.

           // Get notifications from retries from  Transient Fault Handling Application Block code

                retryPolicy.Retrying += (sender1, args) =>

                {

                    // Log details of the retry.

                    var msg = String.Format("Retry - Count:{0}, Delay:{1}, Exception:{2}",

                        args.CurrentRetryCount, args.Delay, args.LastException);

                    //Logging the notification details to the application trace.

                    Trace.Write(msg);

                };

                

                DataCache m_cache = CacheUtil.GetCacheConfig();

     

                try

                {

                    //In order to use the retry policies, strategies defined in Transient Fault Handling Application Block , user calls to cache must be wrapped with in ExecuteAction delegate

                    retryPolicy.ExecuteAction(

                        () =>

                        {

                           //I'm just storing simple string object here .. Assuming this call fails, this sample retries the same call 3 times with 1 second interval before it gives up.

                            m_cache.Put("MyDataSet", "StoreString");

                            //Getting the object from azure cache and printing it on the page.

                            Response.Write(m_cache.Get("MyDataSet"));

                        });

                                    

                }

                catch (DataCacheException dc)

                {

                    // Exception occurred after implementing the Retry logic.

                    //Ideally you should log the exception to your application logs and show user friendly error message on the webpage.

                    Trace.Write(dc.GetType().ToString() + dc.Message.ToString() + dc.StackTrace.ToString());

                }

     

     


Download the sample source code here

 

References

Windows Server AppFabric Caching - Table of Error Code Strings
http://msdn.microsoft.com/en-us/library/ff921008.aspx

Exception Handling Overview (Windows Server AppFabric Caching) – Also applicable to Azure AppFabric Caching
http://msdn.microsoft.com/en-us/library/ff637738.aspx

Handling Errors
http://msdn.microsoft.com/en-us/library/ff637704.aspx

Application Exceptions (Windows Server AppFabric Caching)
http://msdn.microsoft.com/en-us/library/ff921006.aspx  

Transient Fault Handling
http://msdn.microsoft.com/en-us/library/hh680901(v=PandP.50).aspx

 

Post your comments, feedback using comments section below!