With April 2013 CU we are addressing this special issue and there is no need anymore to use the Critical on Demand hotfix provided in February 2013. Cumulative Updates are better tested than a COD.

 

·         Identifying the issue

·         Preparation to install the hotfix

·         Download the hotfix; Updated 10/Apr/2013

·         Installation of the hotfix

·         Post Installation steps

·         Our current recommendation to fix the issue without the hotfix

 

 

Identifying the issue:

 

Issue got introduced since August 2012 CU for SharePoint Server 2010.

 

After publishing a new page the site load times start to increase dramatically or page requests don’t finish loading at all. You will see the infamous “An expected error has occurred“-error or if the callstack is enabled and custom errors are turned off in the web.config (CallStack="true" / customErrors mode="Off") you will see a "Request timed out." error-message.

 

Errors you will see in the ULS log after a user has published a page and then tries to open the page / site containing the page:

 

System.Data.SqlClient.SqlException: Transaction (Process ID 76) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.   

 at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)   

 at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)   

 at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)   

 at System.Data.SqlClient.SqlDataReader.HasMoreRows()   

 at System.Data.SqlClient.SqlDataReader.ReadInternal(Boolean setTimeout)   

 at Microsoft.SharePoint.SPSqlClient.ExecuteQueryInternal(Boolean retryfordeadlock)   

 at Microsoft.SharePoint.SPSqlClient.ExecuteQuery(Boolean retryfordeadlock)

 

 

Unexpected query execution failure in navigation query, HResult -2146232060. Query text (if available): "BEGIN TRAN DECLARE @abort int SET @abort = 0 DECLARE @EidBase int,@EidHome int SET @EidBase = 0 SET @EidHome = NULL IF @abort = 0 BEGIN EXEC @abort = proc_GetWebNavStructNodeByIds '1CA27B52-5021-4CE1-AE8A-C553F62481C5','26F979ED-9769-4F91-B29C-CC45C798D393',13730 SELECT @wssp0 = @abort END IF @abort <> 0 BEGIN ROLLBACK TRAN END ELSE BEGIN COMMIT TRAN END IF @abort = 0 BEGIN EXEC proc_UpdateDiskUsed '1CA27B52-5021-4CE1-AE8A-C553F62481C5' END "

 

 

An unexpected error occurred while manipulating the navigational structure of this Web.

 

 

Leaving Monitored Scope (PortalSiteMapNode: Populating navigation children for web: /ABCDEF). Execution Time=66179.2079302278

 

PortalSiteMapProvider was unable to fetch children for node

at URL: /ABCDEF, message: An unexpected error occurred while manipulating the navigational structure of this Web., stack trace:  

 at Microsoft.SharePoint.SPGlobal.HandleComException(COMException comEx)   

 at Microsoft.SharePoint.Library.SPRequest.UpdateNavigationNode(String bstrUrl, Int32 lNodeId, DateTime dateParented, String bstrName, String bstrNodeUrl, Object& pvarProperties, String& pbstrDateModified)   

 at Microsoft.SharePoint.Navigation.SPNavigationNode.Update()   

 at Microsoft.SharePoint.Publishing.Navigation.SPNavigationSiteMapNode.UpdateSPNavigationNode(SPNavigationNode node, SPNavigationNode previous, String name, String url, String description, String target, String audience, Boolean forceCreate)   

 at Microsoft.SharePoint.Publishing.Navigation.SPNavigationSiteMapNode.UpgradeDraftSPNavigationNode(SPNavigationNode draftNode, SPNavigationNode previous)   

 at Microsoft.SharePoint.Publishing.Navigation.PortalWebSiteMapNode.<>c__DisplayClass2.<UpdateNavigationNodes>b__0()   

 at Microsoft.Office.Server.Utilities.Security.SecurityUtilities.RunWithAllowUnsafeUpdates(SPWeb web, Action secureCode)   

 at Microsoft.SharePoint.Publishing.CmsSecurityUtilities.RunWithAllowUnsafeUpdates(SPWeb web, CodeToRun secureCode)   

 at Microsoft.SharePoint.Publishing.Navigation.PortalWebSiteMapNode.PopulateNavigationChildrenInner(NodeTypes includedTypes)   

 at Microsoft.SharePoint.Publishing.Navigation.PortalWebSiteMapNode.PopulateNavigationChildren(NodeTypes includedTypes)   

 at Microsoft.SharePoint.Publishing.Navigation.PortalSiteMapNode.GetNavigationChildren(NodeTypes includedTypes, NodeTypes includedHiddenTypes, Boolean trimmingEnabled, OrderingMethod ordering, AutomaticSortingMethod method, Boolean ascending, Int32 lcid)   

 at Microsoft.SharePoint.Publishing.Navigation.PortalSiteMapNode.GetNavigationChildren(NodeTypes includedTypes, NodeTypes includedHiddenTypes, OrderingMethod ordering, AutomaticSortingMethod method, Boolean ascending, Int32 lcid)   

 at Microsoft.SharePoint.Publishing.Navigation.PortalSiteMapNode.GetNavigationChildren(NodeTypes includedHiddenTypes)   

 at Microsoft.SharePoint.Publishing.Navigation.PortalSiteMapProvider.GetChildNodes(PortalSiteMapNode node, NodeTypes includedHiddenTypes)

 

 

 

Preparation to install the hotfix:

Check the section “Our current recommendation to fix the issue without the hotfix” to remove unnecessary NavNodes.

  

 

Download the hotfix:

With our April 2013 CU it is now possible to fix this issue. All information about this cumulative update you can find here: http://blogs.msdn.com/b/joerg_sinemus/archive/2013/04/10/sharepoint-2010-and-april-2013-cu.aspx

 

More about the different hotfix packages you can find here: http://blogs.technet.com/b/office_sustained_engineering/archive/2008/07/01/office-hotfixes-to-be-delivered-on-a-defined-schedule-in-the-form-of-cumulative-updates.aspx

   

 

Installation of the hotfix:

As usual like all our cumulative updates we had in the past, see the linked April 2013 CU post.

 

Post Installation Steps:

In case you want to be sure that no additional NavNodes entries are in the database please run the PowerShell script described in the section “Our current recommendation to fix the issue without the hotfix”.

 

 

 

Our current recommendation to fix the issue without the hotfix:

SharePoint 2010 Navigation deadlock: Multiple concurrent requests to update navigation nodes end up in deadlock

- Steps before you install the hotfix

- Steps you need one time after the hotfix has been installed

In case you run into the “NavNodes” issue which is expected only on a Farm with heavy load and lot of changes when using publishing features. One idea is to let the Administrator know which Page creates the unnecessary NavNodes and the other idea is to run a PowerShell script to remove the unnecessary NavNodes.

Our recommendation is to run the PowerShell script one time before you install the hotfix in the farm and one time after the hotfix has been installed. In case you can wait for April 2013 CU you can also use these workarounds until that time.

1.     Regularly run a scheduled powershell script to automatically detect problematic pages (checks all pages on all sites/subsites of a web application) and automatically remove the duplicates 1-by-1 via Powershell using supported and publically SharePoint API.

Ø  DeleteDupNodes_Quicklaunch_and_TopNavigation.txt

=> This will log to a logfile as you can see in the sript… the logging is not perfect – for every node deletion one line in the logfile will be written – logfile will be huge…. This has optimization potential !!!

ð  Some customers report that the deletion of the duplicates just takes too long 30min to 120+min and cannot make use of this workaround and use the next one:

 

2.     Regularly run a scheduled powershell script to automatically detect navigation duplicates and send a mail via SMTP to the admins reporting the problematic pages

Ø  Sendmail_for_duplicates_quicklaunch_and_topnavigation.txt

ð  The admins then manually move the page to the recycle bin and restore them afterwards.

ð  Due to the site is already completely inaccessible you have to use SP Designer or the Manage Content and Structure wizard (sitemanager.aspx) which doesn’t load the navigation.

 

ð  The yellow marked parts have to be correctly set by the customer before running the script:

function sendMail

{

      param ([string]$problemPage)

     

      $emailFrom = "youraddress@yourdomain.com"

      $emailTo = "youraddress@yourdomain.com"

      if($global:navType -eq "TopNavigation")

      {

            $subject = "ALERT: Duplicate navigation-nodes in TopNavigation found!"

      }

      else

      {

            $subject = "ALERT: Duplicate navigation-nodes in Quicklaunch navigation found!"

      }    

      $body = "Problematic page: " + ($problemPage)

      $smtpServer = "YourSMTPServer"

      $smtp = new-object Net.Mail.SmtpClient($smtpServer)

      $smtp.Send($emailFrom, $emailTo, $subject, $body)

}

 

 

 

 

File: DeleteDupNodes_Quicklaunch_and_TopNavigation.txt

 

# THIS CODE-SAMPLE IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED

# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR

# FITNESS FOR A PARTICULAR PURPOSE.

#

# This sample is not supported under any Microsoft standard support program or service.

# The script is provided AS IS without warranty of any kind. Microsoft further disclaims all

# implied warranties including, without limitation, any implied warranties of merchantability

# or of fitness for a particular purpose. The entire risk arising out of the use or performance

# of the sample and documentation remains with you. In no event shall Microsoft, its authors,

# or anyone else involved in the creation, production, or delivery of the script be liable for

# any damages whatsoever (including, without limitation, damages for loss of business profits,

# business interruption, loss of business information, or other pecuniary loss) arising out of

# the use of or inability to use the sample or documentation, even if Microsoft has been advised

# of the possibility of such damages.

################################################################################################

 

 

param

(

    $url = $(Read-Host -Prompt "SiteCollection Url")

)

Add-PSSnapin Microsoft.SharePoint.PowerShell -ErrorAction SilentlyContinue

 

$Logfile = "duplicates_log.txt"

 

Function LogWrite

{

   Param ([string]$logstring)

 

   Add-content $Logfile -value $logstring

}

 

 

function tryDeleteNode

{

    param

    (

        $node,$dictionary,$nodeCollection

    )

 

    $title = $node.Title

   

               

                    if(!$dictionary.ContainsKey($title))

                    {

                        $dictionary.Add($node.Title,$node.Url)

                    }

                    else

                    {

                        if($dictionary[$title] -eq $node.Url)

                        {

                            if($node.Children.Count -eq 0)

                            {

                                echo "       -> Deleting Duplicate Node: $title"

                                $nodeCollection.Delete($node)

                                $global:didDelete= $true

                                               $temp = (get-date).ToString() +";"+ ($site.Url) +";"+ ($title)

                                               echo "$temp"

                                               LogWrite $($temp)

 

                            }

                            else

                            {

                                echo "       -> Dupe Node $title has children, Skipping..."

                            }

                        }

                else

                        {

                    echo "       -> Duplicate title $title found, but mismatched link, Skipping..."

                        }

                    }

               

}

 

function deleteNodesRecurse

{

    $nodes = @{}

    foreach($node in $quickLaunch)

    {

        $childNodes = @{}

        foreach($child in $node.Children)

        {

            tryDeleteNode -node $child -dictionary $childNodes -nodeCollection $node.Children

        }

 

        tryDeleteNode -node $node -dictionary $nodes -nodeCollection $quickLaunch

    }

}

function deleteGlobalNodesRecurse

{

    $nodes = @{}

    foreach($node in $gnavNodes)

    {

        $childNodes = @{}

        foreach($child in $node.Children)

        {

            tryDeleteNode -node $child -dictionary $childNodes -nodeCollection $node.Children

        }

 

        tryDeleteNode -node $node -dictionary $nodes -nodeCollection $gnavNodes

    }

}

 

$sitecoll = Get-SPSite $url

write-host "SiteCollection: " $sitecoll.URL

foreach ($site in $sitecoll.AllWebs)

{

        write-host " -> Site: " $site.URL

                do

                {

                               $quickLaunch = $site.Navigation.QuickLaunch

                               $global:didDelete = $false

                               deleteNodesRecurse

                               $pub= [Microsoft.SharePoint.Publishing.PublishingWeb]::GetPublishingWeb($site)

                               $gnavNodes = $pub.Navigation.GlobalNavigationNodes;

                               deleteGlobalNodesRecurse

                }

                while($global:didDelete)

                $site.Dispose()

}

$sitecoll.Dispose()

 

 

 

 

File: Sendmail_for_duplicates_quicklaunch_and_topnavigation.txt

 

# THIS CODE-SAMPLE IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED

# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR

# FITNESS FOR A PARTICULAR PURPOSE.

#

# This sample is not supported under any Microsoft standard support program or service.

# The script is provided AS IS without warranty of any kind. Microsoft further disclaims all

# implied warranties including, without limitation, any implied warranties of merchantability

# or of fitness for a particular purpose. The entire risk arising out of the use or performance

# of the sample and documentation remains with you. In no event shall Microsoft, its authors,

# or anyone else involved in the creation, production, or delivery of the script be liable for

# any damages whatsoever (including, without limitation, damages for loss of business profits,

# business interruption, loss of business information, or other pecuniary loss) arising out of

# the use of or inability to use the sample or documentation, even if Microsoft has been advised

# of the possibility of such damages.

################################################################################################

 

 

param ($url = $(Read-Host -Prompt "WebApplication Url"))

 

Add-PSSnapin Microsoft.SharePoint.PowerShell -ErrorAction SilentlyContinue

new-variable -name duplicatecounter -visibility public -scope global

new-variable -name navType -visibility public -scope global

new-variable -name duplicateThreshold -visibility public -scope global -value 50

 

function sendMail

{

                param ([string]$problemPage)

               

                $emailFrom = "youraddress@yourdomain.com"

                $emailTo = "youraddress@yourdomain.com"

 

                if($global:navType -eq "QuickLaunch")

                {

                               $subject = "ALERT: Duplicate navigation-nodes in Quicklaunch navigation found!"        

               

                }

                else

                {

                               $subject = "ALERT: Duplicate navigation-nodes in TopNavigation found!"

                }             

                $body = "Problematic page: " + ($problemPage)

                $smtpServer = "YourSMTPServer"

                $smtp = new-object Net.Mail.SmtpClient($smtpServer)

                $smtp.Send($emailFrom, $emailTo, $subject, $body)

}

 

 

function tryDeleteNode

{

                param($node,$dictionary,$nodeCollection)

               

                $title = $node.Title

                if(!$dictionary.ContainsKey($title))

                {

                               $dictionary.Add($node.Title,$node.Url)

                }

                else

                {

                               if($dictionary[$title] -eq $node.Url)

                               {

                                               if($node.Children.Count -eq 0)

                                               {

                                                               $global:duplicatecounter++

                                                               if($global:duplicatecounter -gt $global:duplicateThreshold)

                                                               {

                                                                              $affectedPage = ($site.Url) +"/"+ ($title) + ".aspx"

                                                                              echo "    -> Problem page found: $affectedPage"

                                                                              try

                                                                              {

                                                                                              echo "     -> Sending mail..."

                                                                                              sendMail $affectedPage

                                                                                              echo "     -> Mail sent!"

                                                                              }

                                                                              catch

                                                                              {

                                                                                              echo "     -> error sending Mail!"

                                                                              }

                                                                              $global:didDelete = $false

                                                               }

                                                               else

                                                               {

                                                                              $global:didDelete = $true

                                                               }

 

                                               }

                                               else

                                               {

                                                               echo "    -> Dupe Node $title has children, Skipping..."

                                               }

                               }

                               else

                               {

                                               echo "    -> Duplicate title $title found, but mismatched link, Skipping..."

                               }

                }

 

}

 

function deleteNodesRecurse

{

                $nodes = @{}

                foreach($node in $quickLaunch)

                {

                               $childNodes = @{}

                               foreach($child in $node.Children)

                               {

                                               if($global:duplicatecounter -lt ($global:duplicateThreshold+1))

                                               {

                                                               tryDeleteNode -node $child -dictionary $childNodes -nodeCollection $node.Children

                                               }

                               }

                               if($global:duplicatecounter -lt ($global:duplicateThreshold+1))

                               {

                                               tryDeleteNode -node $node -dictionary $nodes -nodeCollection $quickLaunch

                               }

                }

}

 

function deleteGlobalNodesRecurse

{

    $nodes = @{}

    foreach($node in $gnavNodes)

    {

        $childNodes = @{}

        foreach($child in $node.Children)

        {

                               if($global:duplicatecounter -lt ($global:duplicateThreshold+1))

                               {

                                               tryDeleteNode -node $child -dictionary $childNodes -nodeCollection $node.Children                                

                               }

                }

                if($global:duplicatecounter -lt ($global:duplicateThreshold+1))

                {

                        tryDeleteNode -node $node -dictionary $nodes -nodeCollection $gnavNodes

                }

    }

}

 

$webApplication = Get-SPWebApplication $url

$global:duplicatecounter = 0

foreach($siteCollection in $webApplication.Sites)

{

                write-host "SiteCollection: " $sitecollection.URL

                foreach ($site in $siteCollection.AllWebs)

                {

                               write-host " -> Site: " $site.URL

                               write-host "   ->  Checking QuickLaunch navigation..."

                               $global:navType = "QuickLaunch"

                               do

                               {

                                               $global:didDelete = $false

                                               try

                                               {

                                                               $quickLaunch = $site.Navigation.QuickLaunch

                                               }

                                               catch

                                               {

                                                               write-host "    ->  No navigation nodes..."

                                               }

                                               try

                                               {

                                                               deleteNodesRecurse

                                               }

                                               catch

                                               {

                                               }

                               }

                               while($global:didDelete)

 

 

                               $global:duplicatecounter = 0                    

                               $global:didDelete = $true

 

                               write-host "   -> Checking TopNavigation..."

                               $global:navType = "TopNavigation"

                               do

                               {             

                                               $global:didDelete = $false

                                               $pub= [Microsoft.SharePoint.Publishing.PublishingWeb]::GetPublishingWeb($site)

                                               try

                                               {

                                                               $gnavNodes = $pub.Navigation.GlobalNavigationNodes;

                                               }

                                               catch

                                               {

                                                               write-host "    -> No navigation nodes..."

                                               }

                                               try

                                               {

                                                               deleteGlobalNodesRecurse

                                               }

                                               catch

                                               {

                                               }

 

                               }

                               while($global:didDelete)

 

                               $site.Dispose()

                               $global:duplicatecounter = 0

                               $global:didDelete = $true

                }

                $siteCollection.Dispose()

}