Pradeep Anchan's Blog

SharePoint Upgrade failed with following error. "Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding"

I have come across this scenario several times while troubleshooting the cases where Upgrade process fails with the below error logged in the upgrade.log file:

[AddSearchToLinks] [3.0.77.0] [DEBUG] [6/12/2009 10:11:48 PM]: Executing SQL DDL Script.

[SPContentDatabaseSequence] [ERROR] [6/12/2009 10:41:53 PM]: Action 3.0.77.0 of Microsoft.SharePoint.Upgrade.SPContentDatabaseSequence failed.

[SPContentDatabaseSequence] [ERROR] [6/12/2009 10:41:53 PM]: Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.

[SPContentDatabaseSequence] [ERROR] [6/12/2009 10:41:53 PM]:    at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)

In the above error, we see that the "Executing SQL DDL Script" has been initiated. If you look at the timestamp, it is 6/12/2009 10:11:48 PM. In the next frame, we see the Timeout error at 6/12/2009 10:41:53 PM. The timeout has happened exactly after 30 minutes (That's the timeout period for the thread in the upgrade process to wait)

When I looked into the SQL server, I could see that there was a blocking SPID.

You can use the Activity Monitor in the SQL management studio or simply query the sys.sysprocesses table in the master database as shown below to see if there is any blocking SPIDs.

Use master

select * from sys.sysprocesses

In this particular case, the blocking SPID was coming from a 3rd party Antivirus scanner. After we killed it explicitly, the upgrade continued and completed successfully.

Important Note: The solution above of killing the blocking SPID was allowed in the above specific scenario. This may not be always viable as it depends on what that SPID is doing and whom does it belong. So, in case you see similar issue, it totally depends on your situation and environment how to proceed on it.

Cheers,

Pradeep

Build to Build (B2B) Upgrade of the SharePoint environment – Part 3

In this blog post, we will discuss some tips and techniques around the B2B upgrade process.

Detaching and attaching the Content Database:

Many times during the upgrade process, we may see errors that are specific to one or two content databases especially when you have several content databases and of large size in the farm. In these scenarios, we can detach the specific content database, do the upgrade by running configuration wizard on the farm and then attach the database back. When we attach it back, the content database will be upgraded. Of course, we can encounter the same error, but it will be easier to troubleshoot and also the entire farm will not be down while we are finishing the upgrade of that content database.

Below is the command for detaching the content database:

stsadm -o deletecontentdb -url <url> -databasename <name of the database>

The command for attaching back the content database:

stsadm -o addcontentdb -url <url> -databasename <name of the database>

Orphan entries:

If you encounter any error in upgrade log that indicates orphan entries, for example, "Cannot insert the value NULL into column 'ScopeId'", "The system cannot find the path specified" at the Microsoft.SharePoint.Library.SPRequestInternalClass.GetSiteQuota() method etc, we can check for the existence of orphan entries.

To check for Orphans, use the below command:

stsadm -o databaserepair -url <url> -databasename <name of content database>

To remove/delete the orphan entries, we can run the following command:

stsadm -o databaserepair -url <url> -databasename <name of content database> -deletecorruption

The upgrade process seems to be hung:

Sometimes, depending on the size and the state of the content database, it may take quite a long time for the upgrade process to complete. We should NOT assume it is hung and we should NOT cancel the upgrade process. We need to verify if there is any CPU utilization on the WFE or the SQL box. We can also use SQL Profiler to monitor and check if there is any activity going on the SQL server for the SharePoint databases. We can also keep monitoring the upgrade.log to see if we are progressing or it is really hung/blocked.

Cancelling the upgrade process explicitly in between could leave the farm in a corrupted state. If you are running the upgrade process on the production environment, and if it seems like hung, you can contact Microsoft support line to open a support case for assistance.

While concluding, I want to stress on the point, always test the upgrade on a test environment before doing it on the production environment. Take a Full farm backup from SharePoint level and also take SQL level backup of all the SharePoint databases before doing anything on the production farm. Take the backup of any customizations you have done. Also, make sure that you plan some maintenance window as it may take quite a long time for the upgrade to finish depending on the size and state of the databases and just in case if we encounter any problem, we have enough time to troubleshoot it.

Hope it helps :)

Build to Build (B2B) Upgrade of the SharePoint environment – Part 2

In this post, we will discuss few troubleshooting techniques when the upgrade fails to complete.

The first tool you have in hand is the upgrade.log file. When we run the Configuration Wizard, a file called upgrade.log file will be created, by default in the path C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\LOGS\

As a tip, always make sure you rename the previous upgrade.log file to something so that it creates a new one. Otherwise, it will append to the existing file, making it huge and not very convenient to search for errors.

If the upgrade fails, search for the errors or exceptions in the upgrade.log file. This should give us an idea on what the problem is and the troubleshooting steps will depend on it. We could then use the well known search engines like BING and search for the error and exceptions that we see.

Some of the basic and common issues and possible solutions are listed in KB 944267. Please go through and see if it matches with the error/scenario you have come across. http://support.microsoft.com/?id=944267

I am listing more scenarios and the possible solution here:

1.
Behavior/Symptoms:
===================
The Configuration Wizard Fails at step 8 with the following message:

"Failed to start service SPSearchServiceInstance on this server after completing upgrade. Please start it manually"

In the Upgarde.log file, you may see one of the following errors:

[SPManager] [ERROR] [7/6/2009 5:57:56 PM]: Upgrade [SPSearchDatabase Name=WSS_Search_SPWEB Parent=SPDatabaseServiceInstance] failed.
[SPManager] [ERROR] [7/6/2009 5:57:56 PM]: The service did not start due to a logon failure

Resolution:
===========
The logon username and password may be out of sync (If the password has changed) for the Service Account of the Windows SharePoint Services Search Service.

Go to Central Administration Site -> Services on Server -> Windows SharePoint Services Search, and reset the password for the user.
From the windows services panel, restart the Windows SharePoint Services Search service.

Re-run the configuration wizard by running the following the command

psconfig -cmd upgrade -inplace b2b -force

Additional Note: Refer the KB 934838 to learn how to update the service accounts and passwords for SharePoint.

2.
Behavior/Symptoms:
=================
The Configuration Wizard Fails with the following message:

"Value cannot be null." Parameter name: connectionString

Resolution:
===========
Check if the following registry key exists in the registry:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\12.0\Secure\ConfigDB

"dsn"="<connection string goes here....>"

If dsn doesn't exist, then we can add it manually as a work around and re-run the Configuration Wizard using the following command:

psconfig -cmd upgrade -inplace b2b -wait -force

Below is a sample that can be added.

"dsn" = "Data Source=<db server name>;Initial Catalog=<config db name>;Integrated Security=;Enlist=False"

However, this is just a quick work around step. If registry keys related to SharePoint are missing, we cannot really say if this is the only key missing and how many places has it been tampered. So, the best approach is to disconnecting the machine from the farm and then re-adding it back using the Configuration Wizard.

Important Note: Serious problems might occur if you modify the registry incorrectly. Therefore, please make sure that you back up the registry before you modify it. Then, you can restore the registry if a problem occurs.

3.
Behavior/Symptoms:
=================
The Configuration Wizard Fails with the following message in the PSCDiagnostics log file:
 
Failed to secure the SharePoint resources.

An exception of type System.ArgumentException was thrown. Additional exception information: Cannot apply security settings to resource

C:\WINDOWS\System32\drivers\etc\HOSTS as it does not exist.

System.ArgumentException: Cannot apply security settings to resource

C:\WINDOWS\System32\drivers\etc\HOSTS as it does not exist.
at
Microsoft.SharePoint.PostSetupConfiguration.ResourceAccess.ValidateResourceName(Stri
ng name)
at Microsoft.SharePoint.PostSetupConfiguration.ResourceAccess..ctor(String
resourceType, String resourceName, Int32 securityGroup, String permissions)
at Microsoft.SharePoint.PostSetupConfiguration.SecurityTask.SecureResources()
at Microsoft.SharePoint.PostSetupConfiguration.SecurityTask.Run()
at Microsoft.SharePoint.PostSetupConfiguration.TaskThread.ExecuteTask()

Resolution:
===========
This is pretty straight forward error message and the HOSTS file will be missing from "C:\WINDOWS\System32\drivers\etc\".
We can simply copy a HOSTS file from another machine from the path "C:\WINDOWS\System32\drivers\etc\" and change or add the entries as needed and then re-run the command to perform the upgrade.

4.
Behavior/Symptoms:
=================
The Configuration Wizard Fails at step 4 with the following message:

An exception of type Microsoft.SharePoint.Administration.SPUpdatedConcurrencyException was thrown.

Additional exception information: An update conflict has occurred, and you must re-try this action. The object DiagnosticsService Parent=SPFarm
Name=<config DB> is being updated by Domain\User, in the w3wp process, on machine <machine name>. View the tracing log for more information about the conflict.

Microsoft.SharePoint.Administration.SPUpdatedConcurrencyException: An update conflict has occurred, and you must re-try this action. The object DiagnosticsService Parent=SPFarm Name=<config DB> is being updated by Domain\User, in the w3wp process, on machine <machine name>. View the tracing log for more information about the conflict.
 

Resolution:
===========
Clear the config cache. Refer KB 939308 for the steps for clearing the cache.ini

Re-run the following command to perform the upgrade:

psconfig -cmd upgrade -inplace -b2b -force -wait

5.
Behavior/Symptoms:
==================
The Configuration Wizard Fails with the following message:

Failed to upgrade SharePoint Products and Technologies

An exception of type Microsoft.SharePoint.PostSetupConfiguration.PostSetupConfigurationTaskException was thrown.

Additional exception information: Failed to upgrade SharePoint Products and Technologies.

Observed following error message from the upgrade.log file.

[SPIisWebSiteWssSequence] [ERROR] [7/10/2009 10:19:58 PM]: Action 3.1.3.0 of Microsoft.SharePoint.Upgrade.SPIisWebSiteWssSequence failed.
[SPIisWebSiteWssSequence] [ERROR] [7/10/2009 10:19:58 PM]: The system cannot find the path specified.
[SPIisWebSiteWssSequence] [ERROR] [7/10/2009 10:19:58 PM]: at System.DirectoryServices.DirectoryEntry.Bind(Boolean throwIfFail)

at System.DirectoryServices.DirectoryEntry.Bind()
at System.DirectoryServices.DirectoryEntry.get_AdsObject()
at System.DirectoryServices.PropertyValueCollection.PopulateList()
at System.DirectoryServices.PropertyValueCollection..ctor(DirectoryEntry entry,
String propertyName)
at System.DirectoryServices.PropertyCollection.get_Item(String propertyName)
at Microsoft.SharePoint.Administration.SPIisApplicationPool.get_PeriodicRestartMemory()
at Microsoft.SharePoint.Administration.SPProvisioningAssistant.EnableMemoryBasedAppPoolRecycling(SPIisApplicationPool local)
at Microsoft.SharePoint.Administration.SPProvisioningAssistant.EnableMemoryBasedAppPoolRecycling(String applicationPoolId)
at Microsoft.SharePoint.Upgrade.EnableMemoryBasedAppPoolRecycling.Upgrade()
at Microsoft.SharePoint.Upgrade.SPActionSequence.Upgrade()

Resolution:
=========
Check if the Application Pools are missing in the IIS manager or if the SharePoint Sites are pointed manually to different application Pools. The system administrators might have changed the application pools of the web application as part of the administration tasks.

Note: Since SharePoint stores the Web Application and Application pool settings in its configuration database, Change or deleting the Application pool manually from IIS manager leaves orphan entries in the configuration database. This could impact (For example, the above upgrade issue) later while SharePoint looks into IIS metabase for the Application pools. We can use the SPS Farm reports to view the list of all Web Applications and their associated application pools. The SharePoint Central Administration site doesn't provide any UI to change or delete the application pools.

There is couple of ways to handle the situation.

1. Create/Modify the Application pools manually in IIS manager to bring everything back to how it was before someone changed it. This could be tedious if there are too many.

OR

2. We can delete the web application (Of course, WITHOUT deleting the content DB) in question and re-create it and attach the existing content DB. This will clear the orphan entry.

OR

3. Stop and restart the "Windows SharePoint Services Web Application" service on the WFE. This step will remove and recreate all the application pools and web applications on that WFE, including the missing ones as per what is present in SharePoint configuration datbase.

Imp Note: Make sure you backup the IIS Metabase before trying the above step.

4. Use the object model to programmatically change the web application's associations with the Application pools.
My colleague Vijay has a good blog post on this option. Please refer it for the details and the sample code.

Change the application pool associated with a web application in SharePoint.
http://blogs.msdn.com/vijgang/archive/2009/07/21/change-the-application-pool-associated-with-a-web-application-in-sharepoint.aspx

And then re-run the following command to perform the upgrade:

psconfig -cmd upgrade -inplace -b2b -force -wait

Hope this helps!

In the Part 3 of this blog, I will discuss some more tips and issues on the same topic. Continue to have fun :)

Build to Build (B2B) Upgrade of the SharePoint environment – Part 1

Applying Hot fixes, Cumulative updates (CU) or Service Packs (SP) for WSS 3.0 and MOSS 2007 and then upgrading the databases is a very common scenario. We troubleshoot issues around these areas quite frequently at Microsoft GTSC. I have heard these questions several times from our customers “Which machine do I start running the psconfig wizard in my farm” and “Should we install both WSS and MOSS binaries” etc….

So, I am listing the simple instructions with some recommendations for B2B (Build to Build) Upgrade of the SharePoint environment in Part 1 of this blog post.

Here are the steps:

1. Take a Full SharePoint backup of the entire farm using Central Administration Site. For more information on taking Full backup from SharePoint level, see http://technet.microsoft.com/en-us/library/cc262946.aspx

Note: It is also recommended that you take frequent SQL level backups of the SharePoint databases as part of your disaster recovery plan.

2. On each machine on your farm, install the binaries (Hot fixes, Cumulative updates or Service Packs). In the end of the installation of binaries, you will be prompted to run the Configuration Wizard. Do not run the wizard at this stage.

Note: If you are running MOSS, make sure you install the binaries for both WSS and MOSS. I have seen several cases where people install only WSS update binaries and run the configuration wizard. This is not recommended. If your farm is running only on WSS, then install only WSS binaries.

3. Go to the machine in the farm where you have the Central Administration Site hosted. Run the “SharePoint Products and Technologies Configuration Wizard”. When you initiate the Configuration process, you are shown a message box to run upgrade wizard on all servers in your farm. Click on OK button. Let the Wizard complete. This may take several minutes to several hours depending on the size and the content of the SharePoint databases.

4.  Go to any other machine in the farm and run the “SharePoint Products and Technologies Configuration Wizard”. If it prompts, click Ok and continue to let it finish.

Note: We can also upgrade using the command line tool. The command is as follows:
 psconfig –cmd upgrade –inplace b2b –wait –force

5. Repeat the step 4 on all other machines in the farm one by one. If everything goes fine, the upgrade would be completed.

Hope this helps!

In the Part 2 of this blog, I will discuss some troubleshooting stuff related to B2B upgrade. Till then, have fun J

Cheers,

Pradeep

Kick start - My web blog

Thank you for visiting my Blog!

Although my blog was opened a couple of years ago, wasn't very active. I have now decided to actively blog and share my experiences and ideas. I am working at Microsoft as a Support Escalation Engineer in SharePoint team. I have had plenty of experience on ASP.NET/IIS too. So, I would be concentrating (But not limited) mainly on SharePoint, ASP.NET and IIS. I welcome all your comments, questions and appreciate you valued feedback and suggestions.

Have fun :)

Pradeep Anchan

The SSL is not working on the IIS 5.0 web server. We are not able to browse using https. SSLDiag tool shows "#WARNING:Failed to read schannel data" and "#WARNING: AcquireCredentialsHandle failed with error -2146893051(0x80090305)"

ENVIRONMENT:
==============
Windows 2000 Server SP4, IIS 5.0

AJAX 1.0: "Microsoft JScript runtime error: '0.cells' is null or not an object" error in AJAX enabled ASP.NET page.

CAUSE:

- The DIV contained an UpdatePanel with the id “MultiView”. Inside the
ContentTemplate of the UpdatePanel there was a asp:menu Control.
- The Menu Control has these items:
<Items>
<asp:MenuItem Text="Notes" Value="0" ></asp:MenuItem>
<asp:MenuItem Text="Attachment" Value="1"></asp:MenuItem>
<asp:MenuItem Text="Histroy" Value="2"></asp:MenuItem>
</Items>

- Upon Clicking on any of the above three Menu items , we get “0.cells is null or
not an element” .
- Since the menu Control is not supported in a Partial page Update Scenario , We
will not be able to make the asp:menu control work from
within an UpdatePanel.

WORK AROUND:
The only workaround is to remove the asp:menu Control from within the
ContentTemplate of the UpdatePanel.

- Further reference about controls compatible with the UpdatePanel .
http://www.asp.net/ajax/documentation/live/overview/UpdatePanelOverview.aspx

The operation has timed out. System.Net.WebResponse GetWebResponse(System.Net.WebRequest) at System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest request)

We get the following error when a .NET based Windows Service makes a call to ASP.NET Webservice to upload the file.

The operation has timed out

System.Net.WebResponse GetWebResponse(System.Net.WebRequest)

at System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest request) at  System.Web.Services.Protocols.HttpWebClientProtocol.GetWebResponse(WebRequest request) at  System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters) at  ImageX.Services.RemoteUpload.Client.RemoteUploadWebService.RemoteUploadWebService.InsertFile(String  Physician_ID, String fileName, Byte[] pdfFile, String fType) at  ImageX.Services.RemoteUpload.Client.ixRemoteUploadWindowsService.ProcessQueue()

SOLUTION:

==========

- We need to increase the timeout setting of the web service proxy class before making the call to the web service.

Example:

      localhost.Service1 ws = new localhost.Service1();

      ws.Timeout = 300000;

      ws.HelloWorld();

The Timeout is set in milliseconds for synchronous calls to the XML Web service. The default is 100000 milliseconds. Setting the Timeout property to Timeout.Infinite indicates that the request does not time out; however, the Web server can still cause the request to time out on the server side.

NOTE:

======

- In these kind of scenarios we also need to increase the maxRequestLength in the web.config file for the webservice to match the size of the upload file.

- Also we need to make sure to set the maxconnection setting in the config file for the Windows service/application to appropriate number depending on the way requests are made to the webservice.

More Information:

==============

Improving Web Services Performance

http://msdn2.microsoft.com/en-us/library/ms998562.aspx

Unable to create and debug ASP.NET web application using Visual Studio 2005 on IIS 7.0

How to retrieve the HTML response within a ASP.NET Page

How to retrieve the HTML response generated by ASP.NET pages using HTTP module

ASP.NET Tab is missing in the IIS MMC in the web server running Windows XP Pro x64 Edition with .NET Framework 2.0 (WOW64)

After installing .NET Framework 2.0 on Windows XP Pro x64 Edition, you don't see the ASP.NET Tab in IIS MMC. This happens if IIS is running in WOW64 mode. In other words, if IIS is running in 32 bit mode on a 64 bit Windows operating system, the ASP.NET tab will not appear in the IIS MMC. This is a known issue.

To check whether IIS is configured to run 32 bit applications on 64 bit OS, open the command prompt and navigate to C:\Inetpub\AdminScripts folder and run the following command in the command prompt:

cscript.exe adsutil.vbs get W3SVC/AppPools/Enable32BitAppOnWin64

WORK AROUND:
===============

1. If you are planning to run only ASP.NET 2.0 applications and not ASP.NET 1.1 applications, don't run IIS in 32 bit mode on a server running 64 bit Windows operating system.

To turn off IIS from running 32 bit applications, run the following command

cscript.exe adsutil.vbs set W3SVC/AppPools/Enable32BitAppOnWin64 “false”

2. If you need both ASP.NET 1.1 and 2.0 on 64 bit OS, then you have to enable IIS to run 32 bit applications. In this case the ASP.NET tab will be missing in IIS MMC.

To configure the virtual directories to use specifc ASP.NET versions, you will have to use the command line tool.

aspnet_regiis -s <path of the application>
(Install scriptmaps for this version at the specified path, recursively. Existing
scriptmaps of lower version are upgraded to this version.)
E.g. aspnet_regiis.exe -s W3SVC/1/ROOT/SampleApp1

OR

aspnet_regiis -sn <path of the application>
(Install scriptmaps for this version at the specified path, non-recursively.
Existing scriptmaps of lower version are upgraded to this version.)

 

Thanks,

Pradeep