Recently I was working on an issue and found the worker role was keep recycling. After I logged into Azure VM and looked as Application Event Log, I found the following exception was logged every 30 seconds:
Log Name: Application Source: .NET Runtime Date: 7/6/2011 9:39:28 PM Event ID: 1026 Task Category: None Level: Error Keywords: Classic User: N/A Computer: <Machine_Name> Description: Application: WaWorkerHost.exe Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: System.Data.SqlClient.SqlException Stack: at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(System.Data.CommandBehavior, System.Data.SqlClient.RunBehavior, Boolean, Boolean) at System.Data.SqlClient.SqlCommand.RunExecuteReader(System.Data.CommandBehavior, System.Data.SqlClient.RunBehavior, Boolean, System.String, System.Data.Common.DbAsyncResult) at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(System.Data.Common.DbAsyncResult, System.String, Boolean) at System.Data.SqlClient.SqlCommand.ExecuteNonQuery() at <custom_reference>.SqlErrorLog.Log(Elmah.Error) at <custom_reference>.WorkerRole.GenException(System.Exception) at <custom_reference>.WorkerRole.DequeueMessages() at <custom_reference>.WorkerRole.Run() at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.StartRoleInternal() at Microsoft.WindowsAzure.ServiceRuntime.Implementation.Loader.RoleRuntimeBridge.<StartRole>b__1() at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at System.Threading.ThreadHelper.ThreadStart()
Investigation and Solution:
After Investigation, what we found is that SQL Azure database was set to a max size of 1GB, and that database was full. Some of the worker roles were attempting to write trace data to the SQL Azure DB on startup, and since the database wasn’t accepting new data, the roles terminated unexpectedly. After making room in SQL Azure DB the problem was resolved.