The Web Role was stuck as below:

6:09:56 PM - Preparing...
6:09:56 PM - Connecting...
6:09:59 PM - Uploading...
6:11:15 PM - Creating...
6:12:28 PM - Starting...
6:13:20 PM - Initializing...
6:13:21 PM - Instance 0 of role AvkashTestWebRole is initializing
6:18:39 PM - Instance 0 of role AvkashTestWebRole is busy
6:21:51 PM - Instance 0 of role AvkashTestWebRole is stopped
6:21:51 PM - Warning: All role instances have stopped
6:22:23 PM - Instance 0 of role AvkashTestWebRole is busy
6:23:26 PM - Instance 0 of role AvkashTestWebRole is stopped
6:23:26 PM - Warning: All role instances have stopped

 

I logged into the Azure VM I saw exception logged in Application event log and BootStrapper logs

Following exception related with WaHostBootStrapper.exe was the first in list:

Log Name: Application
Source: Application Error
Date: 6/29/2011 7:40:42 PM
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: RD00155D4224C7
Description:
Faulting application WaHostBootstrapper.exe, version 6.0.6002.18009, time stamp 0x4d6ac763, faulting module ntdll.dll, version 6.0.6002.18446, time stamp 0x4dcd9861, exception code 0xc0000005, fault offset 0x000000000001b8a1, process id 0xbe8, application start time 0x01cc36946462e8a0.

Following BootStrapper log shows the error with WaHostBootStrapper.exe which is stuck as below

[00001744:00001280, 2011/06/29, 19:57:47.150, 00040000] Process exited with 0.
[00001744:00001280, 2011/06/29, 19:57:47.150, 00010000] <- WaitForProcess=0x0
[00001744:00001280, 2011/06/29, 19:57:47.150, 00010000] <- ExecuteProcessAndWait=0x0
[00001744:00001280, 2011/06/29, 19:57:47.150, 00010000] -> WapRevertEnvironment
[00001744:00001280, 2011/06/29, 19:57:47.150, 00010000] <- WapRevertEnvironment=0x0
[00001744:00001280, 2011/06/29, 19:57:47.150, 00010000] -> WapModifyEnvironment
[00001744:00001280, 2011/06/29, 19:57:47.150, 00010000] -> WapUpdateEnvironmentVariables
[00001744:00001280, 2011/06/29, 19:57:47.150, 00010000] -> WapUpdateEnvironmentVariable
[00001744:00001280, 2011/06/29, 19:57:47.150, 00010000] <- WapUpdateEnvironmentVariable=0x0
[00001744:00001280, 2011/06/29, 19:57:47.150, 00010000] <- WapUpdateEnvironmentVariables=0x0
[00001744:00001280, 2011/06/29, 19:57:47.150, 00010000] -> WapGetEnvironmentVariable
[00001744:00001280, 2011/06/29, 19:57:47.150, 00100000] <- WapGetEnvironmentVariable=0x800700cb
[00001744:00001280, 2011/06/29, 19:57:47.181, 00010000] <- WapModifyEnvironment=0x0
[00001744:00001280, 2011/06/29, 19:57:47.181, 00040000] Executing Startup Task type=2 cmd=E:\plugins\RemoteAccess\RemoteAccessAgent.exe
[00001744:00001280, 2011/06/29, 19:57:47.181, 00010000] -> ExecuteProcess
[00001744:00001280, 2011/06/29, 19:57:47.181, 00040000] Executing E:\plugins\RemoteAccess\RemoteAccessAgent.exe.
[00001744:00001280, 2011/06/29, 19:57:47.181, 00010000] <- ExecuteProcess=0x0
[00001744:00001280, 2011/06/29, 19:57:47.181, 00010000] -> WapRevertEnvironment
[00001744:00001280, 2011/06/29, 19:57:47.181, 00010000] <- WapRevertEnvironment=0x0
[00001744:00001280, 2011/06/29, 19:57:47.181, 00010000] -> WapModifyEnvironment
[00001744:00001280, 2011/06/29, 19:57:47.181, 00010000] -> WapUpdateEnvironmentVariables
[00001744:00001280, 2011/06/29, 19:57:47.181, 00010000] -> WapUpdateEnvironmentVariable
[00001744:00001280, 2011/06/29, 19:57:47.181, 00010000] <- WapUpdateEnvironmentVariable=0x0
[00001744:00001280, 2011/06/29, 19:57:47.181, 00010000] <- WapUpdateEnvironmentVariables=0x0
[00001744:00001280, 2011/06/29, 19:57:47.181, 00010000] -> WapGetEnvironmentVariable
[00001744:00001280, 2011/06/29, 19:57:47.181, 00100000] <- WapGetEnvironmentVariable=0x800700cb
[00001744:00001280, 2011/06/29, 19:57:47.228, 00010000] <- WapModifyEnvironment=0x0
[00001744:00001280, 2011/06/29, 19:57:47.228, 00040000] Executing Startup Task type=0 cmd=E:\plugins\RemoteAccess\RemoteAccessAgent.exe /blockStartup
[00001744:00001280, 2011/06/29, 19:57:47.228, 00010000] -> ExecuteProcessAndWait
[00001744:00001280, 2011/06/29, 19:57:47.228, 00010000] -> ExecuteProcess
[00001744:00001280, 2011/06/29, 19:57:47.228, 00040000] Executing E:\plugins\RemoteAccess\RemoteAccessAgent.exe /blockStartup.
[00001744:00001280, 2011/06/29, 19:57:47.228, 00010000] <- ExecuteProcess=0x0
[00001744:00001280, 2011/06/29, 19:57:47.228, 00010000] -> WaitForProcess

Following event log entry shows the error is related with some of the DLL are missing: 

Log Name: Application
Source: ASP.NET 4.0.30319.0
Date: 6/29/2011 7:49:54 PM
Event ID: 1310
Task Category: Web Event
Level: Warning
Keywords: Classic
User: N/A
Computer: Machine_Name
Description:
Event code: 3008
Event message: A configuration error has occurred.
Event time: 6/29/2011 7:49:54 PM
Event time (UTC): 6/29/2011 7:49:54 PM
Event ID: ************************
Event sequence: 1
Event occurrence: 1
Event detail code: 0

Application information:
Application domain: /LM/W3SVC/********/*****************
Trust level: Full
Application Virtual Path: /
Application Path: E:\sitesroot\0\
Machine name: <Machine_Name>

Process information:
Process ID: 2788
Process name: w3wp.exe
Account name: NT AUTHORITY\NETWORK SERVICE

Exception information:
Exception type: ConfigurationErrorsException
Exception message: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective)
at System.Web.Configuration.CompilationSection.LoadAllAssembliesFromAppDomainBinDirectory()
at System.Web.Configuration.AssemblyInfo.get_AssemblyInternal()
at System.Web.Compilation.BuildManager.GetReferencedAssemblies(CompilationSection compConfig)
at System.Web.Compilation.BuildManager.CallPreStartInitMethods()
at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException)

Exception of type 'System.OutOfMemoryException' was thrown.
at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)
at System.Reflection.Assembly.Load(String assemblyString)
at System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective)



Request information:
Request URL: http:// <Internal_IP_Address>/default.aspx
Request path: /default.aspx
User host address: <Internal_IP_Address>
User:
Is authenticated: False
Authentication Type:
Thread account name: NT AUTHORITY\NETWORK SERVICE

Thread information:
Thread ID: 6
Thread account name: NT AUTHORITY\NETWORK SERVICE
Is impersonating: False
Stack trace: at System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective)
at System.Web.Configuration.CompilationSection.LoadAllAssembliesFromAppDomainBinDirectory()
at System.Web.Configuration.AssemblyInfo.get_AssemblyInternal()
at System.Web.Compilation.BuildManager.GetReferencedAssemblies(CompilationSection compConfig)
at System.Web.Compilation.BuildManager.CallPreStartInitMethods()
at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException)

Launching web application using local IP address in the Virtual Machine I see the following error:

Server Error in '/' Application. 
________________________________________
Exception of type 'System.OutOfMemoryException' was thrown.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

Source Error:
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Stack Trace:

[OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.]
System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks) +0
System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection, Boolean suppressSecurityChecks) +567
System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection) +192
System.Reflection.Assembly.Load(String assemblyString) +35
System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective) +118

[ConfigurationErrorsException: Exception of type 'System.OutOfMemoryException' was thrown.]
System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective) +11396867
System.Web.Configuration.CompilationSection.LoadAllAssembliesFromAppDomainBinDirectory() +484
System.Web.Configuration.AssemblyInfo.get_AssemblyInternal() +127
System.Web.Compilation.BuildManager.GetReferencedAssemblies(CompilationSection compConfig) +334
System.Web.Compilation.BuildManager.CallPreStartInitMethods() +280
System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException) +1087

[HttpException (0x80004005): Exception of type 'System.OutOfMemoryException' was thrown.]
System.Web.HttpRuntime.FirstRequestInit(HttpContext context) +11529072
System.Web.HttpRuntime.EnsureFirstRequestInit(HttpContext context) +141
System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context) +4784373

________________________________________
Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.0.30319.1

Note:

The key here is that “Out of memory exception could mislead” which is actually caused because of “some reference DLL is missing” at first place. See the exception hierarchy to understand it. 

Solution:

A few references were not set to "Copy Local as true" and after setting all necessary references to "Copy Local as True" and then upgrade the deployment solved the problem.