Unpredictably, intermittently, and frequently clients trying to open office documents (word, excel, powerpoint, etc) in the browser from the SharePoint 2010 farm using office web apps get an error in the browser instead of the document.
Sometimes the error includes a correlation ID and sometimes it doesn't.
An examples of the error messages are:
Word Web App cannot open this document for viewing because of an unexpected error. To view this document open it in Microsoft Word. Error ID: ….. OK
PowerPoint Web App cannot open this presentation for viewing because of an unexpected error. To view this presentation, open it in Microsoft PowerPoint.
No corrective action needs to be taken for the problem to go away. Refreshing the page several times is usually enough to get the document to open in OWA.
The documents do open fine in the Office client.
Confirmed in Central Admin that the Word Viewing Service (for example) is actually running on the expected SharePoint servers.
Examining the httperr logs (c:\windows\system32\logfiles\httperr) showed no good clues. There were, for example, no interesting mentions of cellstorage.svc.
Renaming one of the offending office documents doesn't change the behavior.
Verbose ULS logging while reproducing the problem turned up these interesting bits of data:
04/15/2014 10:43:49.76 w3wp.exe (0x1A04) 0x48A8 Office Web Apps Office Viewing Architecture b4vw Medium RequestDispatcher: dispatching request for document F6f53802479144af8b60b8aafe6051580m486879ca364a4e15bf6e56464c0527adm1b304927e3e04ccb85575838653825f1m to http://web1:32843/e313c22d07334006870adf2aaccf2d45/Conversion.svc. 33bf84b3-f74f-4a7f-a75b-884dab5e4dc6
04/15/2014 10:43:49.78 w3wp.exe (0x2754) 0x0F54 Office Web Apps Office Viewing Architecture bv6p Unexpected Exception thrown downloading file C:\Windows\TEMP\waccache\e313c22d-0733-4006-870a-df2aaccf2d45\TREASURYECM_SpServices\3d42c3c8-45b5-413a-874a-f31e9ea69bcc\output.docx: System.ArgumentException: SharepointReaderAsync: no SPFile at Microsoft.Office.Web.Environment.Sharepoint.SharepointReaderAsync.End() at Microsoft.Office.Web.Conversion.Framework.DownloadManager.OnFileReadComplete(IAsyncResult ar) 33bf84b3-f74f-4a7f-a75b-884dab5e4dc6
04/15/2014 10:43:49.78 w3wp.exe (0x2754) 0x0F54 Office Web Apps Office Viewing Architecture fvj9 Unexpected DownloadManager: Download FAILED for document F6f53802479144af8b60b8aafe6051580m486879ca364a4e15bf6e56464c0527adm1b304927e3e04ccb85575838653825f1m; Time spent: 2ms. 33bf84b3-f74f-4a7f-a75b-884dab5e4dc6
The following are suggestions to consider. As usual, standard disclaimers apply.
First, per http://support.microsoft.com/kb/2521084, one common cause for this symptom is that "the application pool account was unable to access the content database" and that a possible solution is "we changed the application pool account password and synced it with sharepoint" or to give the service account DBOwner rights on the content database. "Make sure the application pool account and word viewing service account have dbo access to the content databases."
It might also be a good idea to check the application event log(s) for these types of messages:
Log Name: Application Source: Microsoft-SharePoint Products-SharePoint Foundation Event ID: 3760 Task Category: Database Level: Critical Description: SQL Database 'db_.........WSS_Content' on SQL Server instance '…..' not found. Additional error information from SQL Server is included below. Cannot open database "db……WSS_Content" requested by the login. The login failed. Login failed for user '…'.
Some people have said that just changing the account of that the application pool and/or the word viewing service (or powerpoint viewing service or other OWA service) to the farm account temporarily and then changing it back to the original service account helps.
Also you may need to use the sysinternals Process Monitor to find "access denieds" on temp folders.
Second, one suggestion is to ensure that the content database has been updated to match the SharePoint version. Please check the database version level and dll versions per the steps below:
Go to Central Admin > Operations > Servers in farm.
Note version number and send to me please.
(This is the version of the databases.)
Open this folder on all the SharePoint servers: c:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\ISAPI (ßThe 14 hive there assumes this is
Look for OWSSVR.dll and Microsoft.SharePoint.Portal.dll
Please send me the file version number on each.
If the versions don't match up, I'd recommend running either psconfigUI.exe (the configuration wizard) or psconfig.exe (psconfig –cmd upgrade –inplace b2b –wait).
Third, ensure that the following folders and processes are excluded from real-time-antivirus-scanners:
Other RTAV exclusion information can be found here: http://blogs.msdn.com/b/chaun/archive/2013/08/01/do-we-really-need-to-set-antivirus-exclusions-up-for-our-sharepoint-servers.aspx
Fourth, sometimes these errors are judged to be caused by WAC Cache Corruption. If so, the solution here is to rebuild the WAC cache per http://blogs.technet.com/b/wbaer/archive/2010/09/01/the-office-web-applications-cache.aspx. Excerpt:
Clearing the Cache
To clear the Office Web Applications cache you will need to clear the server file system cache and the Site Collection cache for the Web application.
1. Delete the cache files from the server file system cache
2. Delete the cache files from the site collection cache
Maybe try running the Office Web Apps Expiration Timer job first.
Fifth, t might be a good idea to look into the question of whether or not the hardware load-balancer uses sticky sessions of some kind or not.