I recently had a request to provide a SharePoint Administrator the ability to export out a crawl log to a CSV file using PowerShell.  Luckily, I found Vijay’s (Thanks Vijay!) post which saved me a ton of time:

http://blogs.msdn.com/b/spses/archive/2011/06/22/exporting-sharepoint-2010-search-crawl-logs.aspx

I wanted to add some more functionality to this process of exporting out to a CSV.   I wrote the following PowerShell script which gives the Admin more options like choosing which Search Service Application and filtering by URL or by Content Source.

Updated on 2/18/2012 – Thanks to my colleague Heiko at Microsoft.  He pointed out the lack of functionality when multiple crawl stores exists within the Search Service Application.  I updated the script to now work with Search Service Applications containing multiple crawl stores.

This script does the following: 

1.    Let you decide which Search Service Application to use
2.    Let you decide if you want to filter against URL or by Content Source
3.    If you choose URL, you have an option of only exporting Errors or All Events  (Skip to Step 5)
4.    If you choose Content Source, you decide which content source to filter against
        4.a. After Choosing Content Source, you have an option of only exporting Errors or All Events

5.    Finally, you can choose the path and file name to use and the file will export the result set as a CSV which can be opened  in Excel

Instructions for running the script:  
 
1.    Copy the below script and save it in notepad
2.    Save it with a anyfilename.ps1 extension
3.    To run, copy the file to a SharePoint Server
4.    Select Start\Microsoft SharePoint 2010 Products\SharePoint 2010 Management Shell
5.    Browse to directory holding the copied script file
6.    Run the script:  .\anyfilename.ps1 (assuming anyfilename is the name of the file)

<# ==============================================================
//
// Microsoft provides programming examples for illustration only,
// without warranty either expressed or implied, including, but not
// limited to, the implied warranties of merchantability and/or
// fitness for a particular purpose.
//
// This sample assumes that you are familiar with the programming
// language being demonstrated and the tools used to create and debug
// procedures. Microsoft support professionals can help explain the
// functionality of a particular procedure, but they will not modify
// these examples to provide added functionality or construct
// procedures to meet your specific needs. If you have limited
// programming experience, you may want to contact a Microsoft
// Certified Partner or the Microsoft fee-based consulting line at
// (800) 936-5200.
//
// For more information about Microsoft Certified Partners, please
// visit the following Microsoft Web site:
// https://partner.microsoft.com/global/30000104
//
// Author: Russ Maxwell (russmax@microsoft.com)
//
// ----------------------------------------------------------  #>

[Void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint")

Start-SPAssignment -Global

#############################
#Function to Export the Data#
#############################
function exportThis
{
    $output = Read-Host "Enter a location for the output file (For Example: c:\logs\)"
    $filename = Read-Host "Enter a filename"
    $stores = $ssa.CrawlStores
    $storectr = $stores.count
    $name = $output + "\" + $filename + ".csv"
   
    if($storectr -eq '1')
    {
        $logViewer = New-Object Microsoft.Office.Server.Search.Administration.Logviewer $ssa
        $i = 0
        $urlOutput = $logViewer.GetCurrentCrawlLogData($crawlLogFilters, ([ref] $i))
        Write-Host "# of Crawl Entries Produced" $urlOutput.Rows.Count
        $urlOutput | Export-Csv $name -NoTypeInformation
        Write-Host "Your results were exported to: " $name
    }
   
    elseif($storectr -gt '1')
    {
        $f = 1
        foreach($store in $stores)
        {
          Write-Host "In the " $f " iteration of store object"
          $logViewer = New-Object Microsoft.Office.Server.Search.Administration.Logviewer $store
          $i = 0
          $urlOutput = $logViewer.GetCurrentCrawlLogData($crawlLogFilters, ([ref] $i)) 
          $ctr = $urlOutput.Rows.Count
          $officialCTR += $ctr
                 
            if($f -eq '1')
              {
                $finalDT = New-Object System.Data.DataTable
                $finalDT = $urlOutput.Copy()
              }
              else
              {
               $finalDT.Merge($urlOutput)
              }
         
          $f++
         
          }
    }       
          $finalDT | Export-Csv $name -NoTypeInformation         
          Write-Host "# of Crawl Entries Produced" $officialCTR
}

#####################################
#Choose a Search Service Application#
#####################################
$ssa = Get-SPEnterpriseSearchServiceApplication
$ssaName = $ssa | ForEach-Object {$_.Name}
Write-Host "Choose a Search Service Application to review crawl logs"
Write-Host
$num = 1

Foreach($sa in $ssa)
{ Write-Host $num $sa.Name
  $num++
}

Write-Host
$result = Read-Host "Enter the number next to the desired Search Service Application and press enter"

$num = 1
Foreach($i in $ssa)
{
if($num -eq $result)
{
    $ssa = $i
}
$num++
}
Write-Host

###############################################
#Create a Logviewer and Crawl Log FilterObject#
###############################################
$crawlLogFilters = New-Object Microsoft.Office.Server.Search.Administration.CrawlLogFilters

######################
#Let the Admin choose#
######################
Write-Host "How would you like to filter the crawl log?"
Write-Host "1 Filter Based on a URL"
Write-Host "2 Filter Based on Content Source"
Write-Host "3 Export without a Filter"
Write-Host
$choice = Read-Host "Enter 1, 2, or 3 and press enter"
Write-Host
Write-Host
Write-Host "1 Export only errors"
Write-Host "2 Export All (Success, Warning, and Errors"
$type = Read-Host "Enter 1 or 2 and press enter"
Write-Host

if($choice -eq '1')
{
                $url = Read-Host "Enter the URL to filter on"
               
                ###################################
                #Create Property and add to filter#
                ###################################
                $totalentryProp = New-Object Microsoft.Office.Server.Search.Administration.CrawlLogFilterProperty
                $totalentryProp = [Microsoft.Office.Server.Search.Administration.CrawlLogFilterProperty]::TotalEntries
                $urlProp = New-Object Microsoft.Office.Server.Search.Administration.CrawlLogFilterProperty
                $urlProp = [Microsoft.Office.Server.Search.Administration.CrawlLogFilterProperty]::Url
                $stringOp = New-Object Microsoft.Office.Server.Search.Administration.StringFilterOperator
                $stringOp = [Microsoft.Office.Server.Search.Administration.StringFilterOperator]::Contains
                $crawlLogFilters.AddFilter($urlProp, $stringOp,$url)
                $crawlLogFilters.AddFilter($totalentryProp, "1,000,000")
               
                if($type -eq '1')
                {
                                $typeEnum = New-Object Microsoft.Office.Server.Search.Administration.MessageType
                                $typeEnum = [Microsoft.Office.Server.Search.Administration.MessageType]::Error
                                $crawlLogFilters.AddFilter($typeEnum)
                }
                               
                #Calling exportThisfunction
                exportThis
}

elseif($choice -eq '2')
{
               
                #########################
                #Choose a content source#
                #########################
                $content = New-Object Microsoft.Office.Server.Search.Administration.Content($ssa)
                $contentsources = $content.ContentSources
               
                Write-Host "Choose a Content Source to filter on"
                Write-Host
                $num = 1

                Foreach($c in $contentsources)
                {
                                Write-Host $num": " $c.Name
                                $num++
                }

                $result = Read-Host "Enter the associated # press enter"

                $num = 1
                Foreach($c in $contentsources)
                {
               if($num -eq $result)
               {
                  $contentSource = $c
               }
               $num++
                }
                Write-Host "You chose" $contentSource.Name
                $id = $contentSource.Id
               
               
                ###################################
                #Create Property and add to filter#
                ###################################
                $totalentryProp = New-Object Microsoft.Office.Server.Search.Administration.CrawlLogFilterProperty
                $totalentryProp = [Microsoft.Office.Server.Search.Administration.CrawlLogFilterProperty]::TotalEntries
                $csProp = New-Object Microsoft.Office.Server.Search.Administration.CrawlLogFilterProperty
                $csProp = [Microsoft.Office.Server.Search.Administration.CrawlLogFilterProperty]::ContentSourceId
                $crawlLogFilters.AddFilter($csProp, $id)
                $crawlLogFilters.AddFilter($totalentryProp, "1,000,000")
               
                if($type -eq '1')
                {
                                $typeEnum = New-Object Microsoft.Office.Server.Search.Administration.MessageType
                                $typeEnum = [Microsoft.Office.Server.Search.Administration.MessageType]::Error
                                $crawlLogFilters.AddFilter($typeEnum)
                }
               
                #Calling exportThisfunction#
                exportThis
               
}   

elseif($choice -eq '3')
{
                $catProp = New-Object Microsoft.Office.Server.Search.Administration.CatalogType
                $catProp = [Microsoft.Office.Server.Search.Administration.CatalogType]::PortalContent
                $crawlLogFilters.AddFilter($catProp)
                $catProp2 = New-Object Microsoft.Office.Server.Search.Administration.CatalogType
                $catProp2 = [Microsoft.Office.Server.Search.Administration.CatalogType]::ProfileContent
                $crawlLogFilters.AddFilter($catProp2)
                $totalentryProp = New-Object Microsoft.Office.Server.Search.Administration.CrawlLogFilterProperty
                $totalentryProp = [Microsoft.Office.Server.Search.Administration.CrawlLogFilterProperty]::TotalEntries
                $crawlLogFilters.AddFilter($totalentryProp, "1,000,000")
               
                if($type -eq '1')
                {
                                $typeEnum = New-Object Microsoft.Office.Server.Search.Administration.MessageType
                                $typeEnum = [Microsoft.Office.Server.Search.Administration.MessageType]::Error
                                $crawlLogFilters.AddFilter($typeEnum)
                }
               
                #Calling exportThisfunction
                exportThis
}
    
Stop-SPAssignment –Global

Thanks!

Russ Maxwell, MSFT  

I apologize it’s been so long since I’ve published any new blog content.  I recently moved into a new position in Microsoft as a Premier Field Engineer.  I’m really enjoying it so far and will start pumping out new blog content soon.  For now, I recently wrote a PowerShell script which fetches the item count for items that reside in all Document Libraries for a given Site Collection.  That includes all Document Libraries that exists in both the root site and all sub sites. 

The script will produce the following output:

1.    Total # of Document Libraries with 0 items
2.     Total # of Document Libraries with 1 or more items
2.    Total # of items in all Document Libraries
3.    Total # of items in DocLib\Subfolders
4.     Start Time and End Time of Script

Instructions for running the script:  
 
1.    Copy the below script and save it in notepad
2.    Save it with a anyfilename.ps1 extension
3.    To run, copy the file to a SharePoint Server
4.    Select Start\Microsoft SharePoint 2010 Products\SharePoint 2010 Management Shell
5.    Browse to directory holding the copied script file
6.    Run the script:  .\anyfilename.ps1 (assuming anyfilename is the name of the file)

Script is below:

<# ==============================================================
// Microsoft provides programming examples for illustration only,
// without warranty either expressed or implied, including, but not
// limited to, the implied warranties of merchantability and/or
// fitness for a particular purpose.
//
// This sample assumes that you are familiar with the programming
// language being demonstrated and the tools used to create and debug
// procedures. Microsoft support professionals can help explain the
// functionality of a particular procedure, but they will not modify
// these examples to provide added functionality or construct
// procedures to meet your specific needs.
// Author: Russ Maxwell (russmax@microsoft.com)
// ----------------------------------------------------------  #>

[Void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint")
Start-SPAssignment -Global

$starttime = Get-Date
#Creating new site object
$siteurl = Read-Host "Enter the name of your site and press enter"
$site = New-Object Microsoft.SharePoint.SPSite($siteurl)

#Assigning all webs (sites) to $webs
$webs = $site.Allwebs

####################################################
#System Libraries defined so they won't be touched##
####################################################

$systemlibs =@("Converted Forms", "Customized Reports", "Documents", "Form Templates", 
                              "Images", "List Template Gallery", "Master Page Gallery", "Pages", 
                               "Reporting Templates", "Site Assets", "Site Collection Documents",
                               "Site Collection Images", "Site Pages", "Solution Gallery", 
                               "Style Library", "Theme Gallery", "Web Part Gallery", "wfpub")

Write-Host  "Total number of webs that will be traversed: " $webs.count

$DocLibsCount = 0
$DocLibwItems = 0
$totalitems = 0
$subfolderitems = 0

foreach($web in $webs)
    {
        $listcoll = $web.lists
       
        foreach($list in $listcoll)
        {
             if($list -eq $null)
             {
                Write-Host
             }
             else
             {
                $base = $list.GetType()
                if($base.name -eq "SPDocumentLibrary")
                {
                    if ($systemlibs -contains $list)
                    { continue}
                    else
                        { 
                          $DocLibsCount += 1   
                          $items = $list.items
                         
                          if($items -ne "0")
                          {
                           $DocLibwItems += 1
                           Write-Host "Processing ItemCount for DobLib " $DocLibsCount -ForegroundColor Red
                           $totalitems  += $items.count
                  
                           $name = $list.Title
                           $folders  = $web.GetFolder($name).SubFolders
                           for($etr = 0;$etr -lt $folders.count; $etr++)
                            {
                             if($folders[$etr].Name -ne "Forms")
                               {
                                 Write-Host "Processing SubFolder ItemCount" -ForegroundColor Red
                                 $tempcount = $folders[$etr].ItemCount
                                 $subfolderitems += $tempcount
                                 }
                            }
                           }
                          }
                   }
                 }
              }
          } 
 
Write-Host
Write-Host
Write-Host "Total # of Document Libraries: " $DocLibsCount -ForegroundColor Green
Write-Host "Total # of Document Libraries that contain items: " $DocLibwItems -ForegroundColor Green
Write-Host "Total # of items: " $totalitems -ForegroundColor Green
Write-Host "Total # of items in DocLib\Subfolders: " $subfolderitems -ForegroundColor Green
$finishtime = Get-Date
Write-Host
Write-Host “Script Duration” –ForegroundColor Yellow
Write-Host “Started:   “ $starttime –ForegroundColor Yellow
Write-Host “Finished: “ $finishtime –ForegroundColor Yellow

Stop-SPAssignment -Global

Thanks,

Russ Maxwell, MSFT  

Troubleshooting SharePoint Performance issues are probably the most challenging issues for a SharePoint Administrator to troubleshoot.   Collecting Performance Monitor output can take time because not only must you understand the options available in the UI, but you also need to know which counters to add.   Microsoft does have some automation available to assists in this area by using publicly available PLA interfaces.  I wrote a PowerShell script that any SharePoint Administrator can save off to a Web Front-End, run, and start Performance Gathering in a quick and efficient manner.  First, I’d like to give a super huge thanks to Brad Rutkowski.  He tipped me off that this was possible using PowerShell by creating –com objects and leveraging the PLA interfaces.  His blog documenting interacting with data collector sets is here:

http://blogs.technet.com/b/brad_rutkowski/archive/2009/02/18/interacting-with-data-collector-sets-via-powershell.aspx

The publicly available PLA interfaces and MSDN documentation was a great help as well. 

http://msdn.microsoft.com/en-us/library/aa372243(v=VS.85).aspx

Also, the Windows SDK provided some slick samples of how to do this in CPP. 

Reasons why I wrote this script:

1. To write something that was publicly available outside the SharePoint object model
2. To automate a task that takes much longer to implement in the UI
3. To provide more configurable options to the user running the script

In order to run this, save the below script to a text editor like notepad.   Save the file with a .ps1 extension to a SharePoint server.   To run the file from PowerShell, run .\yourfilename.ps1

Important:  I successfully tested this script on both SharePoint 2007 and SharePoint 2010 installed on Windows 2008.  This script will prompt the user to input the maximum size in MB for the performance monitor log.  When the maximum size is reached,  a new performance log (file) is created and used.  This cycle continues until the Performance Data Collection is manually stopped.   I confirmed this script can run against remote SharePoint servers which means you won’t be required to run this script on each SharePoint Server.  Rather, you can run it from one SharePoint server and for each time you run the script, specify the destination SharePoint Server.   I provided an option in the script to allow for automatically starting the Data Collector to immediately start collecting performance data.   Below is a sample run I performed and I set opted to have the script automatically start the data collector called SharePointRocks.  

What the script looks like

Data Collector set automatically started

Script is below.  Enjoy!

#######################
#Grab Initial Settings#
#######################
$Name = Read-Host "Enter a name your Data Collector Set?"
$Machine = Read-Host "Enter the SharePoint Server Name to monitor"
$Sample = Read-Host "Please Enter sample interval in Seconds"
$output = Read-Host "Please enter location for the output (For Example: C:\Perflogs)"
Write-host "Please enter the maximum size of each output file in MB"
$buffer = Read-Host "I recommend 250"

############################
#Create the Data Collector Set#
############################
$DataCollSet = new-object -ComObject pla.DataCollectorSet
$DataCollSet.DisplayName = $Name

#######################
#Create a Data Collector#
#######################
$DataCollector = $DataCollSet.DataCollectors.CreateDataCollector(0)
$DataCollector.name = "SharePoint - Collect"
$filename = $Machine + ".blg"
$DataCollector.FileName = $filename
$DataCollector.FileNameFormat = 0x4000
$DataCollector.SampleInterval = $Sample

######################################################################
#Building up Array of counters and adding as property to the Datacollector#
######################################################################
$counters = @("\ASP.NET v2.0.50727\*","\ASP.NET Apps v2.0.50727(*)\*", "\.NET CLR Networking(*)\*", "\.NET CLR Memory(*)\*", "\.NET CLR Exception(*)\*", "\.NET CLR Loading(*)\*", "\.NET Data Provider for SqlServer(*)\*", "\Processor(*)\*", "\Process(*)\*", "\LogicalDisk(*)\*", "\Memory\*", "\Network Interface(*)\*", "\PhysicalDisk(*)\*", "\Web Service(*)\*", "\Web Service Cache\*", "\System\*", "\TCPv4\*", "\TCPV6\*", "\SharePoint Publishing Cache(*)\*")
$DataCollector.PerformanceCounters = $counters

########################################################################
#Add the data collector to the data collector set and set additional properties#
########################################################################

try
{
    $DataCollSet.DataCollectors.Add($DataCollector)
    $DataCollSet.Segment = "VARIANT_TRUE"
    $DataCollSet.SegmentMaxSize = $buffer
    $DataCollSet.RootPath = $output

    $DataCollSet.Commit($Name, $machine, 0x0003)

    Write-Host "The Data Collector set has been created Successfully." -ForegroundColor Green
   
}
catch [Exception]
    {
      Write-Host "Exception Caught: " $_.Exception -ForegroundColor Red
      return
    }
   
   
Write-Host "Would you like to automatically start the data collection?"
$decide = Read-Host "Press 1 or yes or 2 for no"

if($decide = 1)
{
    try
    {
    $DataCollSet.Start($true)
    Write-Host "Operation Completed Successfully" -ForegroundColor Green
    Write-Host
    }
    catch [Exception]
    {
      Write-Host "Exception Caught: " $_.Exception -ForegroundColor Red
      return
    }
}

else
{
    Write-Host
    Write-Host "To access and start this Data Collector set:"
    Write-Host "1. Select Start, Run, and type Perfmon (press enter)"
    Write-Host "2. Expand Data Collector Sets\User Defined"
    Write-Host "3. Right click on desired data collector set to Start"
}

Write-Host
Write-Host "Script Completed"

Thanks!

Russ Maxwell, MSFT 

Snapshots and SharePoint 2010 are a great thing in that they provide an additional level of fault tolerance for any SharePoint 2010 farm.  A SharePoint Administrator can quickly restore Sites, Lists, or items from a snapshot.   This out of the box solution is possible through the Central Administrators Recover Data from an unattached content database feature.   This functionality exports the user selected data out of the Snapshot database and into a backup file.  Finally, the backup file and be used with import-spweb to restore the data.  For a walkthrough of restoring snapshots using Central Administrator,  see my blog posts here. 

I recently found out this functionality isn’t possible using out of the box PowerShell cmdlets.   I put together a PowerShell script that will guide you through Recovering Data from an Unattached database.   It’s possible to recover from a Site, Subsite, List, or Document Library.  Once, the export is complete, simply run import-spweb cmdlet.   I recommend running this in a test farm to get more familiar with it.  

In order to run this, save the below script to a text editor like notepad.   Save the file with a .ps1 extension.   To run the file from PowerShell, run .\yourfilename.ps1

I assume you know the following things when running this script:

1.  If the end goal is restoring a specific subsite, it’s expected that you know the partial path to your subsite. 

For Example:   Site Collection is:  http://contoso

Let’s assume subsite1 is a subsite that resides directly under contoso and subsite2 is a subsite that resides directly under SubSite1.

The full url looks like:   http://contoso/subsite1/subsite2

• To restore subsite1, the partial url is:  /subsite1
• To restore subsite2, the partial url is: /subsite1/subsite2

2.  It’s expected that you know the list or document library name.  

3.  It’s assumed that your Central Admin site’s display name contains the words “Central Admin”, if this isn’t the case, adjust the $_.DisplayName property on the line which declares $caweb variable.

Enjoy!

[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint")
Add-PSSnapin Microsoft.SharePoint.PowerShell

Start-SPAssignment -Global
Write-Host "This PowerShell script will assists you in exporting data out of a snapshot and into a physical file which can then be restored via import-spweb."  
Write-Host "  "

####################
#    Initial Setup #
####################

#Get the CA URL
$caweb = Get-SPWebapplication -IncludeCentralAdministration | where{$_.DisplayName -match "Central Admin"}
$ca = $caweb.url

##############################
#Manually Grab other settings#
##############################
$SQLserv = Read-Host "Enter your SQL Server Name"
$location = Read-Host "Enter desired backup directory (For Example: D:\backupdir)"
$filename = Read-Host "Enter desired backup filename  (For Example: bufile.cmp)"
$includesec = Read-Host "Press y to include security or any other key to export without security"

##################################################
#Choose a ContentDB and inserting into $contentDB#
##################################################
$cdb = Get-SPContentDatabase 
$cdbname = $cdb | ForEach-Object {$_.Name}
Write-Host
write-host "Choose a Content Database from which the snapshot was taken"
Write-Host
$num = 0

Foreach ($i in $cdbname)
{
    Write-Host $num $i
    $num++
}

$result = Read-Host
$num = 0
Foreach($i in $cdbname)
{
   if($num -eq $result)
   {
       $contentDB = $i
   }
   $num++
}
  

###############################################  
#Select a snapshot and inserting it in $snapdb#
###############################################
$snapdbs = Get-SPContentDatabase $contentDB
$snap = $snapdbs.Snapshots | ForEach-Object {$_.Name}
$num = 0
Write-Host
Write-Host "Select a Snapshot"
Foreach ($i in $snap)
{
    Write-Host $num $i
    $num++
}
$num = 0

$result = Read-Host
Foreach($i in $snap)
{
   if($num -eq $result)
   {
       $snapdb = $i
   }
   $num++
}

#####################
# Grab the Snapshot #
#####################
$snapshot = [Microsoft.SharePoint.Administration.SPContentDatabase]::CreateUnattachedContentDatabase($SQLserv, $snapdb, $null, $null)

################
#Choose a site #
################
$snapnames = $snapshot.Sites | ForEach-Object {$_.Url}
Write-Host
Write-Host "Choose a site to export from"
$num = 0
ForEach($i in $snapnames)
{
    write-host $num $i
    $num++
}
$result = Read-Host
$num = 0
ForEach($i in $snapnames)
{
     if($num -eq $result)
    {
        $URL = $i
    }
    $num++
}

Write-Host "You chose site"
Write-Host $URL

##############################
#Only export from a subsite? #
##############################
write-host "Press y to export from a specific subsite"
write-host "Press any other key to export from the site collection level"
$subdec = Read-Host
if(($subdec -eq "Y") -or ($subdec -eq "y"))
{
        write-host "Please Enter the subsite name in the format of /subsitename"
        write-host "For Example: http://contoso/subsite1 would look like /subsite1"
        $subsite = Read-Host
        $URL2 = $URL + $subsite
}

###############################
#Only export a list or doclib?#
###############################
Write-host "Press y to only export a list or document library"
Write-Host "Press any other key to export from site level"
$listit = Read-Host
if(($listit -eq "y") -or ($listit -eq "Y"))
{
    Write-Host "Please enter the list or doc library in the format of /testlist"
    Write-Host "For Example: http://contoso/testlist would look like: /testlist"
    $list = Read-Host
    if(($subdec -eq "Y") -or ($subdec -eq "y"))
    {
        $URL2 = $URL2 + $list
    }
    elseif(($subdec -ne "Y") -or ($subdec -ne "y"))
    {
        $URL2 = $URL + $list
    }
}

#####################################
#  Inserting Site or List to export #
#####################################
$exportobject = New-Object Microsoft.SharePoint.Deployment.SPExportObject

if(($listit -ne "y") -or ($listit -ne "Y"))
{
    $exportobject.Type = [Microsoft.SharePoint.Deployment.SPDeploymentObjectType]::Web
    if(($subdec -eq "Y") -or ($subdec -eq "y"))
    {
        $exportobject.Url = $URL2
    }
    else {$exportobject.Url = $URL}
}

elseif(($listit -eq "y") -or ($listit -eq "Y"))
{
    $exportobject.Type = [Microsoft.SharePoint.Deployment.SPDeploymentObjectType]::List
    $exportobject.Url = $URL2
}

$exportobject.IncludeDescendants = [Microsoft.SharePoint.Deployment.SPIncludeDescendants]::All

##############################
# Configuring Export Settings#
##############################
$exportsettings = New-Object Microsoft.SharePoint.Deployment.SPExportSettings
$exportsettings.UnattachedContentDatabase = $snapshot
$exportsettings.SiteURL = $URL
$exportsettings.FileLocation = $location
$exportsettings.LogFilePath = $location
$exportsettings.BaseFileName = $filename
$exportsettings.ExportObjects.Add($exportobject)
$exportsettings.IncludeVersions = [Microsoft.SharePoint.Deployment.SPIncludeVersions]::All
$exportsettings.LogExportObjectsTable = 1

if($includesec -eq "y" -or "Y")
{
  $exportsettings.IncludeSecurity = [Microsoft.SharePoint.Deployment.SPIncludeSecurity]::All
}

####################################
#  Create Export Object and Export #
####################################
$export = New-Object Microsoft.SharePoint.Deployment.SPExport($exportsettings)
Write-Host
Write-Host "Starting Export"
$export.Run()
Write-Host
Write-Host "Operation Completed Successfully!"

Stop-SPAssignment –Global

Thanks!

Russ Maxwell, MSFT  

I’m excited to kick this series off because I really think PowerShell rocks. The goal of this series is to provide some automation for SharePoint Administrators to perform various deployment and/or troubleshooting tasks. To use the script simply copy it to notepad and save it with a ps1 extension. To run the script on your SharePoint 2010 environment, simply launch PowerShell and get to the directory where the script is located. To run the script type:

.\scriptname.ps1

The first script was created based on a blog I previously authored here. Please review that blog for more information. The following script provides the following functionality:

Selecting Option 1:

Creates a Content type derived from document and puts it in a special _hidden group at the site level. It also adds the hidden content type to a document library

Selecting Option 2:

If the hidden content type was created previously, choosing option 2 will add the hidden content type

Note: You will need to keep track of the names of the hidden content type by name after they are created.

[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint")

Start-SPAssignment -Global
$siteurl = read-host "Enter the name of your site and press enter"
$site = New-Object Microsoft.SharePoint.SPSite($siteurl)
[Microsoft.SharePoint.SPWeb]$web = $site.OpenWeb()

Write-Host "Press 1 to create a hidden Content Type and add it to a Document Library"
Write-Host "Press 2 to add a previously created hidden Content type to a Document Library"

[int]$result = read-host

if($result -eq 1)
{
       [Microsoft.SharePoint.SPContentTypeCollection]$cts = $web.ContentTypes
       $parent = $web.ContentTypes | where {$_.id -eq "0x0101"}

       $ctName = Read-Host "Enter the name of your Content Type"
      
       #Creating ContentType and adding to the site collection
       $ct = New-Object Microsoft.SharePoint.SPContentType($parent, $cts, $ctName)
       $cts.Add($ct)
      
       #Putting the Content Type in the hidden Group
       $Ct.Group = "_Hidden"
       $ct.Update()
      
       #Adding Content Type to Document library
       Write-host "Adding Hidden Content Type to Document Library"
      
       $doclib = Read-host "Type the name of the Document Library you want to add it to and press Enter key."
       [Microsoft.SharePoint.SPList]$list = $web.Lists[$doclib];
      
       [Microsoft.SharePoint.SPDocumentLibrary]$oDocumentLibrary = $list;
      
       $oDocumentLibrary.ContentTypesEnabled = "true"
       
       write-host "Adding new hidden Content Type to Document Library"
       $oDocumentLibrary.ContentTypes.Add($ct)
       $list.Update()
      
       write-host("Operation Completed Successfully")

}

elseif($result -eq 2)
{
       $hiddenct = Read-Host "Enter the name of the Hidden Content Type and press Enter"
       $doclib = Read-host "Enter the name of the Document Library you want to add it to and press Enter"
      
       #Referencing DocLib
       [Microsoft.SharePoint.SPList]$list2 = $web.Lists[$doclib]
       [Microsoft.SharePoint.SPDocumentLibrary]$oDocumentLibrary = $list2;
       $oDocumentLibrary.ContentTypesEnabled = "true"
      
       #Pull the ContentType from the Hidden Group and Add it to the DocLib
       $cthid = $web.AvailableContentTypes[$hiddenct]
      
       #Add hidden Content Type to DocLibrary
       write-host "Adding hidden Content Type to Document Library"
       $oDocumentLibrary.ContentTypes.Add($cthid);
        $list2.Update();
       write-host "Operation Completed Successfully"
}

else {write-host "Run the script again and choose option 1 or 2"}

#Disposing Objects now
Stop-SPAssignment -Global

Thanks!

-Russmax  

I recently worked on an interesting issue with SharePoint 2010 Content Types. The specific request was the following:

A request to create an additional Content Type in a Document Library that wasn’t visible to other Document Libraries within a Site Collection.

This sounds easy but is a little more complex because when you create a content type at the site collection level, it’s automatically available to all other list and document libraries within the specified site collection. The original request was to prevent the content type from being visible to other Document libraries. The traditional method of simply creating a content type at Site Collection and adding it to preferred Document Library isn’t going to work. The original plan was to only pull from content types available on the Document Library.

The following PowerShell script was used to create an additional content type that’s available on that specific Document Library:

[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint")
$SiteURL= "http://<serverName>"
$Library = "Shared Documents"
$site = new-object Microsoft.SharePoint.SPSite($SiteURL)
$web = $site.OpenWeb()
$lib = $web.Lists[$Library]
$item = $lib.ContentTypes["Item"];
$ct = new-Object Microsoft.SharePoint.SPContentType($item,$lib.ContentTypes,"TestContentType")
$lib.ContentTypes.Add($ct)

This works in that I can only see my newly created content type “TestContentType” within the specified Document Library.

Problem

This exposed a new problem working with word documents and the new content type. The following steps provide more detail:

1. When selecting a new Document, select the new Content Type:

2. Put some random text in the document and save it back to Document Library.

The expectation is that the document is saved using the “new” TestContentType. In this scenario, it doesn’t and uses the Default Document Content type. This is true even if the TestContentType is marked as default. Viewing properties on the Document displays the following:

Looking further at the newly created content type:

In the above screenshot, the Source is Item and no Document Source is present. When creating Word Documents via additional content types, the Content Type used must be derived from base Document content type. If you use a Content Type derived from Item, this by design behavior will occur in Document Libraries.

From MSDN:  http://msdn.microsoft.com/en-us/library/ms463016.aspx

“However, it is important to know that you cannot add every content type that is available in a given site to every list or library in the site. Any content type that you add to a document library must inherit from the built-in Document content type or from a content type that is derived from Document.”

Question: How can I create a content type derived from Document without using the content types available at the Site level?

Answer: You can’t if the Document Library is already created. The Document Content Type is already in use “Default Document Content Type” by default. Some advanced steps make it possible to add additional content types only available to a particular Document Library. However, it’s tedious and outside the scope of this blog.

First, thanks goes out to my colleague Gyorgy at Microsoft for tipping me off about the hidden group. The solution is to create a new Content Type at the site collection level and add it to the Hidden Group. The hidden group is a special group in that any content types moved there will not be visible to any list or document library.

Solution 1: Using the UI

Create a new content type

1. Access Site, Site Actions, and Site Settings
2. Under category Galleries, select Site content types
3. Select Create and Type in a Name for your content type
4. Under Select parent content type from: choose Document Content Types
5. Under Parent Content Type: choose Document
6. Under Existing Group: choose Document Content Type

It should look like this before hitting OK:

 Add new content type to Document Library

1. Access the Document Library and choose Library from the ribbon
2. Select Library Settings, Advanced Settings
3. Ensure that Allow management of content types is set to Yes and hit OK
4. Back to library settings page, under Content Types select “Add from existing site content types”
5. Choose the newly created content type, add, and hit OK

Move Content type into Hidden Group

1. Access Site, Site Actions, and Site Settings
2. Under category Galleries, select Site content types
3. Find and click on the newly created content type
4. Within ManageContentType.aspx select Name, description, and group
5. Select New Group and type _Hidden and hit OK

Note: This is case sensitive so it must look like: _Hidden

After the above steps, the content type is hidden and should not be visible when attempting to add it to new document libraries using the UI. Also, you can’t manipulate the content type within Site Content Types because it’s hidden. Solution 1 is good for one time use.

Question: What if this is a hidden content type that you add it to document libraries after it’s already marked as hidden in the future?

Answer: To add a hidden content type to additional document libraries requires custom code.

Solution 2: Leverage the Object Model

I’m not a big fan of solution 1 because usually the same content type will be added more than once. I decided to write a C# application that can do all of this for you.

Note: If option 2 is selected, you must know the name of the hidden content type so keep all hidden content type names in a safe location you can refer to by name at a later date.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.SharePoint;

namespace Content
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Enter the name of your site and press enter");
            string siteurl = Console.ReadLine();

            using (SPSite site = new SPSite(siteurl))
            {
                using (SPWeb web = site.OpenWeb())
                {

                  //Check to see if we should just create a content type
                 //or move on to add one that's already created

                 Console.WriteLine("Press 1 to create a hidden Content Type and add it to
                                                     a Document Library");

                 Console.WriteLine("Press 2 to add a previously created hidden Content Type
                                                     to a Document Library");
                    
                 int result = int.Parse(Console.ReadLine());

                 
                 if (result == 1)
                    {
                        Console.WriteLine("Enter the name of your ContentType");
                        string ctName = Console.ReadLine();

                        // Create a new site content type.
                        SPContentTypeCollection cts = web.ContentTypes;
                        SPContentType ct = new SPContentType
                        (cts[SPBuiltInContentTypeId.Document],cts, ctName);

                         // Add the content type to the site collection.
                        cts.Add(ct);
                        Console.WriteLine(
                        "Added {0} content type to site collection.", ct.Name);

                        //Put the content type in group _Hidden
                        ct.Group = "_Hidden";
                        ct.Update();

                        Console.WriteLine("Type the name of the Document Library you want
                                                             to add it to and press Enter key.");
                       
                        string doclib = Console.ReadLine();
                        SPList list = web.Lists[doclib];

                        SPDocumentLibrary oDocumentLibrary = (SPDocumentLibrary)list;
                        oDocumentLibrary.ContentTypesEnabled = true;

                        Console.WriteLine("Adding new hidden Content Type to Document
                                                             Library");
                       
                        SPContentType lstCT = oDocumentLibrary.ContentTypes.Add(ct);
                        list.Update();
                        Console.WriteLine("Operation Completed Successfully");
                    }

                    else if (result == 2)
                    {
                        Console.WriteLine("Please enter the Content Type name now");
                        string hiddenct = Console.ReadLine();

                        Console.WriteLine("Enter the name of the Document Library where the 
                                                             Content Type will be added.");
                       
                        string Dirk = Console.ReadLine();

                        SPList list2 = web.Lists[Dirk];
                        SPDocumentLibrary oDocumentLibrary = (SPDocumentLibrary)list2;
                        oDocumentLibrary.ContentTypesEnabled = true;

                        //Pull the Content Type from hidden group
                        SPContentType cthid = web.AvailableContentTypes[hiddenct];

                        //Adding hidden content type to Document Library
                        Console.WriteLine("Adding hidden content type to Document Library");
                        oDocumentLibrary.ContentTypes.Add(cthid);
                        list2.Update();

                        Console.WriteLine("Operation completed successfully");

                    }

                    else
                    {
                       Console.WriteLine("Please run the application again and choose option
                                                           (1 or 2)");
                    }
                 }
            }
        }
    }
}

I'm starting a blog series in which I'll be providing Power Shell scripts that do various things in SharePoint 2010.  The first topic in the PowerShell Scripting series is a script that allows you to create a content type derived from Document and mark it as hidden while adding it to a document library.  The second part of the script will provide the ability to add hidden content types previously created to additonal document libraries.  The script can be pulled from here! 

Thanks,

-RussMax

 The manage access requests feature is great for allowing users that don’t have permissions to a SharePoint site requests access. This link is exposed when users attempt to access a site they don’t have permission to.

Clicking on the link automatically sends an email to the email address specified within Manage Access Requests/Access Request Settings.

Setting up Manage Access Request

In order to setup Manage Access Request, you must have inbound/outbound email setup in SharePoint 2010.

http://technet.microsoft.com/en-us/library/cc263462.aspx#section1

Assuming email is setup, you may go to your preferred site and choose Site Actions, Site Settings, and choose Site permissions under Users and Permissions section. Manage Access Requests is exposed in the ribbon:

Clicking on Manage Access Requests will take you here:

 Usually, the original Site Owner email address is specified here.

Are you sure you want to use Manage Access Requests feature?

I would take a serious look into whether or not Manage Access Requests feature is the right approach for a company with a large SharePoint farm and site collection owners change daily, weekly, or monthly. The assumption here is that a site collection owner email address is specified to receive access request emails. Let’s assume my site collection owner is contoso\admin and the Manage Access Requests email address is set to the same user: Admin@contoso.com. Contoso\Admin took another job within the same company but no longer owns or supports this particular SharePoint site. Contoso\Admin is replaced by Contoso\jrAdmin. Contoso\jrAdmin adds his account to Site Owners group while removing Contoso\Admin account from Site Owners group for this particular SharePoint site.

500 users are hired to this particular company and all are instructed to access this SharePoint site and requests access. 500 emails get sent to Contoso\Admin instead of Contoso\jrAdmin. When removing site owners, it has no effect on the email address specified within Manage Access Request. This can quickly become a nightmare for the company help desk if regular users are site owners and site owners change often in a large SharePoint farm. This behavior is by design for both SharePoint 2007 and SharePoint 2010. If the feature is important for your company and you must use it, I recommend adding technical documentation on how to fully remove site collection owners by adding a section on updating the Manage Access Requests email field to the new site collection owner.

Thanks,

Russ Maxwell, MSFT

We have two main issues with provisioning User Profile Service Application. First, props go out to Sheyi at Microsoft who’s done some extremely valuable work in discovering, troubleshooting, and communicating these issues. Also, I need to give props to Jose at Microsoft for doing some key validation work.

Both of these issues affect usability of the User Profile Service Application. Specifically, Installing SharePoint 2010 and applying the December CU prior to setting up the User Profile Service Application could be affected by this. 

The December CU can be found here:

http://technet.microsoft.com/en-us/sharepoint/ff800847.aspx

Note: This puts SharePoint at build: 14.0.5130.5002

Question: How can I find what build of SharePoint is installed?

Answer: Easiest way is to launch Central Administrator/Operations and click on Servers in Farm link

Before reviewing the two issues I assume that you already have a firm grasps of the steps involved in provisioning User Profile Service Application including the associated Active Directory Synchronization Connection. If not, I recommend reviewing my previous blog here.

Issue 1

This is already documented luckily in kb 2490381 and a work around exists. The symptom is when an attempt to create an Active Directory Sync connection and specifying more than one domain results in the following error:

“Unable to process Create message”

The work around is to simply add one domain and hit OK. Then go back and edit the AD DS synchronization that was just created and select additional domains.

http://support.microsoft.com/kb/2490381/

Issue 2

Attempting to create an Active Directory sync connection when NetBiosDomainNamesEnabled property is set to true produces the following error:

“Unable to process Create message”

There is currently no workaround for this. If you would like to create an Active Directory Synchronization Connection and use NetBiosDomainNamesEnabled property, then do not apply the December CU.

Question: Will it be fixed?

Answers: On a positive note, both issues are fixed in the upcoming February CU.  I don’t have a date of when to expect the release of this CU but it’s coming soon..

Thanks,

Russ Maxwell, MSFT

Recently, some of my coworkers and I started to see some customer problems where membership wasn’t populating for users my sites. Membership population issues are complex and difficult to troubleshoot if you’re not sure where to start looking. Part 1 of this blog is intended to cover the basics of how site Membership population to users My Sites work in SharePoint 2007. I’ll add some troubleshooting tips as well. Part 2 will focus on known Site Membership population issues and resolution to those issues.

Intro to Membership

So what is membership? A perfect intro has already been written so here it is:

“Microsoft Office SharePoint Server 2007 supports two types of memberships: Distribution List (DL) memberships and Windows SharePoint Services site memberships. DL membership information is obtained from the Active Directory directory service, and Windows SharePoint Services site membership information is obtained by pulling membership information from the SharePoint site. A user's public My Site page, called the Profile page, displays the user's memberships, as well as memberships the user and the viewing user have in common, among other information.”

From:

http://msdn.microsoft.com/en-us/library/ms492573(v=office.12).aspx

I’ll focus on SharePoint site membership since it’s the source of all of the issues I’ve encountered recently. The Profile Synchronization timer job is responsible for synching the site membership information from a content database to the SSP database. It has other tasks but this blog will focus on how this job syncs site membership from Content database to SSP database. This timer job runs hourly by default. A membership webpart resides on a user’s mysite page to display both site and group membership:

The requirements to get membership populated are the following:

Note: This assumes that an SSP is already provisioned, a full crawl has been performed, and users my site is provisioned.

1.) Users must reside in the sites default members group (contribute rights)

2.) Create an extra site group with contribute rights (membership of this group doesn’t matter)

3.) Wait for the Profile Synchronization Timer job to run

Behind the Scenes

Before diving into any troubleshooting, it’s important to understand that there are many moving parts for this single timer job. We leverage SQL tables in both the SSP database and Content database. I won’t cover the stored procedure names and what they do because a Profile Spec document already exists and is available publicly here. This is a high level summary of what are the moving parts and how they work together to ensure a user can view membership on his/her mysite.

Change Log

When starting profile sync, the first thing we do is notify the SSP’s Profile Stats table indicating profile sync has started. Next, a decision is made on whether or not to perform a Full or Incremental Sync. The concept is similar to how changes are detected via an Incremental Crawl (Search). In this case, the last change is recorded from the previous sync in the SSP’s ContentDBSynch table. When ProfileSynch timer job runs, we fetch the latest change from this table. We use that to query changes against the associated Content Databases event cache table looking for changes greater than the change log value. The types of changes queried are adds/removes/changes for objects like sites, users, and groups.

What about New Sites?

If new sites are detected, we register those in the SSP’s SiteSync table.

What about Users?

If changes to users (Adds, Modify, Removes) are collected, we subsequently collect the associated User’s SID and Site ID from the Content databases UserInfo table. The user collected data is finally pushed to the SSP’s UserSites table.

What about Groups/Group Membership?

Groups are pulled from the Content databases SiteGroupsView table. GroupMembership is pulled by a special SQL query which pulls this data from both the UserInfo table and Group Membership table. We create some temp tables within the SSP to store the Group and Group Membership data pulled from the Content DB.

Almost Completed

Groups are added from the associated SSP’s temp table to the SSP’s MemberGroups table. Group Membership information is pulled from the associated SSP Temp table and copied into the SSP’s User Memberships table. Next, the SSP’s ContentDBSync table is updated with the latest change log and the SSP Profile_Stats table is updated to change status to complete. Finally, the membercount column within the membergroup table is updated for the particular groups that have added/removed users.

Initial Troubleshooting

These are some recommended initial troubleshooting steps to perform when site membership isn’t populating for users My Site.

Step 1: Crank down the timer job

Troubleshooting why site membership isn’t populating to one or more users mysite is difficult to troubleshoot. OWStimer process is responsible for making this magic happen via the Profile Synchronization timer job. Waiting hourly for this timer job to run and hoping for the best isn’t the best use of time when it simply doesn’t appear to work. The first thing I would do is crank down how often this timer job runs until you get site membership populating. You can do this with the following command:

stsadm -o sync -synctiming M:1

This causes the Profile Synchronization job to run every minute.

Step 2: ULS log with ULS Viewer

I would also crank up verbose ULS logging for categories Timer and User Profiles in Central Administrator/Operations/Diagnostic Logging. Finally, download ULS viewer from here and monitor the ULS logs real time:

http://code.msdn.microsoft.com/ULSViewer

Note: You may need to hop around to different SharePoint servers in a multi-server farm with ULS Viewer. The server that picks up the scheduled timer job first will run it and you’ll see something like the following in the ULS logs:

09/25/2010 12:20:00.15 OWSTIMER.EXE (0x0740) 0x08C4 Windows SharePoint Services Timer 8e45 Verbose Begin invoke timer job Profile Synchronization, id {C516F71C-0018-40AD-AD9F-13453209DE0F}, DB {EC9FAAE6-29A5-4F6D-9CC2-5E8DCAEC0569}

You can use ULS viewer to filter on the event ID 8e45 within your ULS logs to find out which server kicked this timer job off. Once you find this event, then it’s simply a matter of filtering the ULS log based on the thread ID. In this example, the thread ID is 0x08c4.

Start looking for any error/warnings/critical events that fire until timer job has completed. Once you get to the following trace event, the Profile Sync job has completed:

09/25/2010 12:20:01.97 OWSTIMER.EXE (0x0740) 0x08C4 Windows SharePoint Services Timer 8e46 Verbose End invoke timer job Profile Synchronization, id {C516F71C-0018-40AD-AD9F-13453209DE0F}, DB {EC9FAAE6-29A5-4F6D-9CC2-5E8DCAEC0569}

I’ll cover most of the SQL stored procedure calls you find during the profile synch review in the ULS logs in the advanced troubleshooting section.

Step 3 – Make user group changes on the associated site

Ensure that the user or users that are lacking membership presence in his/her mysite is added to the default sites members group. I would also ensure that particular group has contribute rights.

How do I get there?

1. Access the SharePoint site

2. Select Site Actions, Site Settings

3. Select People and Groups

4. Select More link

5. Should see all groups including the default site name Members group

6. Click on sitename members group “test3 Members” and ensure the particular user is added to this group

7. Click on the edit box next to sitename members group “test3 Members” and ensure contribute right is checked

Also, ensure that a custom created SharePoint group is created and has contribute rights. Finally, trying to toggle the user by doing the following is a valid test:

1. Removing the user from sitemembers group (For Example: test3 Members)                                                                                                                                                                                                                                                                                                                                                                                                                                             2. Running the profile sync timer  job                                                                                                                                                                                                                                                                                                                                                   3. Adding the user back to sitemembers group (For Example: test3 Members)                                                                                                                                                                                                                                                                         4. Running the profile sync timer job

This may or may not resolve the issue but this will make more sense why I recommend it after reviewing the advanced troubleshooting section.

Advanced Troubleshooting - (One or two users)

This section is for troubleshooting one or two users. The usual symptom is that that mysite fails to display site membership for one or two users. Every user’s my site displays site membership correctly except for these one or two users.

After the Profile Synch timer job has run

In order to display site membership via mysites, some SQL tables are used to store/retrieve this information. The membergroup table contains site groups that have been sync’d. The UserMemberships table contains the user membership information and references the associated membergroup. To validate that a user\users have properly synched and associated with the appropriate site group, you must query the UserMemberships and Membergroup tables within the SSP database.

For Example, I want to validate the profile synch pushed the Test3 site group to the SSP’s Membergroup table.

select * from Membergroup with(NOLOCK) where DisplayName = 'test3'

There are other columns but I’m looking for the Id of the particular group which is 9 in this case.

The UserMembership table within the SSP database contains all the users and defines what MemberGroup ID they belong to.

The UserMembership table is defined by a SID and MemberGroupID. In this case, we identify the user by SID so it’s not as simple as locating a user by his/her name. Also, you cannot perform select queries based on SID. Luckily the SID information is located in the Content databases UserInfo table. The following sample SQL query requires the following from you:

The red portions of the query are where you manually insert the three above items.

/*Manually replace SSP_DB with the name of your SSP database*/

USE SSP_DB

/*Replace WSS_Content with the name of your Content  database and Replace tp_Login with the specified domain\user*/

select Distinct MemberGroupId from UserMemberships INNER JOIN WSS_Content.dbo.UserInfo ON UserMemberships.SID = WSS_Content.dbo.UserInfo.tp_SystemID and tp_Login = 'domain\user'

Example Output:

This SQL query will output all of the MemberGroupID’s , “Sites”, that a particular user is a member of.

Question: What does this mean?

Answer: The SSP is aware that the user is assigned as a member of a site group. That is, the user resides in the SSP’s UserMembership table and assigned to the appropriate member group (SiteGroup).

Question: How do I know which Site Groups represent a specific MemberGroupId?

Answer: Run the following SQL query:

/*Manually replace SSP_DB with the name of your SSP database*/                                                                                                                                                                                                                                                                                             USE SSP_DB

/*Replace ID with the ID’s with the MemberGroupID from the previous Query*/                                                                                                                                                                                                                                                                       select Id, DisplayName, Url, MemberCount from MemberGroup with(NOLOCK) where Id = '9' OR Id = '10' OR Id = '11'

Before Profile Synch timer job run

Step 1 - Is the user or group changes available for the Profile Synch Timer job?

The Profile Synch timer job has many moving parts. It acts similar to the way incremental crawls (search) works by keeping track of the latest change synchronized from the Content database. It stores this into the SSP’s ContentDBSynch table. We then perform several queries against the Content Databases event cache table to determine and collect changes like the following:

1) Site Group has been modified (added/deleted)

2) Users have been added/removed to Site Group

Note: This isn’t everything but just to point out a few examples

Run the following to get the latest change ID:

--Script to pull current change token--

/*Replace SharePoint_Config with your Config database name*/                                                                                                                                                                                                                                                                                                  use SharePoint_Config                                                                                                                                                                                                                                                                                                                                                                           Declare @DB Uniqueidentifier

/*Replace WSS_Content with the name of your content database*/                                                                                                                                                                                                                                                                                      SELECT @DB = Objects.Id from Objects with(NOLOCK) where Name = 'WSS_Content'

/*Replace SSP_DB with your SSP database name*/                                                                                                                                                                                                                                                                                                                        use SSP_DB                                                                                                                                                                                                                                                                                                                                                                                       select CurrentChangeToken from ContentDBSynch with(NOLOCK) where ContentDBID = @DB

It outputs something similar to the following:

The last 4 digits is the last change SSP is aware of which is 5533. In this example, I want to query all records from event cache table that are greater than 5533 and have the following object types:

• 128 - User
• 1 - Item
• 256 - Group

The following query will confirm that changes were logged to the above three objects with a change greater than 5533.

select * from EventCache with(NOLOCK) where Id > 5533 and ObjectType = '128' or Id > 5533 and ObjectType = '1' or Id > 5533 and ObjectType = '256'

The output looks like:

This is good in that new changes are available after you perhaps added some users to SharePoint site groups etc… However, what does this information mean and how can you use it to your advantage? I would test by performing the following:

1. Add a user to the default site membership group.                                                                                                                                                                                                                                                                                                                          2. Validate the user appears in eventcache with later

I’ve added a user named Jon to site Test 5’s default members group. I know that the change is greater than 5533 from the previous query.

--Change WSS_Content to the name of your Content database--                                                                                                                                                                                                                                                                                               Use WSS_Content

Declare @USER int

--Change 'Domain\User' to the user you tracking--                                                                                                                                                                                                                                                                                                                              SELECT @USER = tp_id from UserInfo with(NOLOCK) where tp_Login = 'Contoso\jon'

select * from EventCache with(NOLOCK) where Id > 5533 and ItemId = @USER

Now I have a more scoped output that contain only entries directly related with User Jon that I added J

That’s it for troubleshooting prior to Profile Synch timer job run for one or two users. It’s important to validate that the changes are available to be picked up by the Profile Synch timer job prior to runtime. It’s equally important to ensure the SSP has logged membership correctly for individual users. We do have a couple of known issues with this specific symptom to be aware of which will be in Part 2 in this blog series.

Thanks!

Russmax

I’ve seen a couple of these issues come from our customers so wanted to get the word out.  

When crawling PPTX files with embedded links in 2007 generates the following error in the crawl logs:

“The filtering process could not be initialized. Verify that the file extension is a known type and is correct". 

.

Before applying the fix, ensure this is the issue you’re running into by reviewing the crawl log and looking at the actual documents to ensure they have embedded links.

.

The fix is applying the Microsoft Office 2010 Filter pack on the 2007 server hosting the Index role:

http://www.microsoft.com/downloads/en/details.aspx?FamilyID=5cd4dcd7-d3e6-4970-875e-aba93459fbee&displaylang=en

.

Thanks,

-Russmax

I ran into a particularly unique issue where any user was unable to create alerts for one discussion board. All other libraries, lists, and discussion boards worked and the same users could create alerts with no problem.

Steps to validate you’re hitting this particular problem:

1. Go to a Document Library, list, or discussion board

2. Create a new alert from a list or discussion board by select Actions/Alert me

3. Fill out the New Alert Page and click OK

Result: Get the following error: “Unknown Error”

This error doesn’t give you enough information. In order to get more information, it’s necessary to enable a couple options in the associated web.config file.

Edit the corresponding web.config file and change the following values:

For Example:

1. Access IIS Manager
2. Right click the appropriate site, choose explore
3. Right click the web.config
4. Open With, NotePad

Should now look like the following:

5. Change the customErrors and CallStack to the following:

CallStack="true"

customErrors mode="Off"

6. Save the web.config and now attempt to reproduce the issue.
7. Now the error should look like:

"Object reference not set to an instance of an object. at Microsoft.SharePoint.ApplicationPages.SubNewEditBasePage.SetAlertProperties(SPAlert a, SPWeb web, SPList list, String strAlertTemplateName, RadioButtonList RadioBtnEventType, RadioButtonList RadioBtnAlertFreq, RadioButtonList RadioBtnAlertFilter, TextBox TextTitle, DropDownList DdlView) at Microsoft.SharePoint.ApplicationPages.SubNewPage.BtnCreateAlert_Click(Object sender, EventArgs e) at System.Web.UI.WebControls.Button.OnClick(EventArgs e) at System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument) at System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(String eventArgument) at System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument)
at System.Web.UI.Page.RaisePostBackEvent(NameValueCollection postData)
at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) "

Cause:

This error occurs because the discussion board, list, or document libraries original dispform.aspx was deleted using SharePoint designer. We have some dependencies on the original dispform.aspx so if this is ever deleted within SharePoint designer, this problem will occur.

Resolution:

The best resolution is to restore the problem discussion board, list, or document library prior to when the deletion occurred. The other method is to move the items to a new library. A variety of options are available to move data over. The object model can be leveraged or the items can be moved over via Manage Content and Structure. While these methods work fine for Lists and Document libraries, discussion boards are a little more challenging. I found the easiest way to move Discussion board items was using Outlook.

Note: Using Outlook to move discussion board items will not preserve the metadata.

Question:

I used SharePoint designer and my custom list is pointed to my new dispform.aspx.   Can I delete the old dispform.aspx which I renamed?

Answer:

Even if SharePoint designer is used to point to a new dispform.aspx, you must keep the original.  You can rename it but it needs to say put.

Thanks,

Russmax

I recently had an interesting issue where the customer was getting the following errors accessing the User Profiles and properties link within his SSP.

“An error has occurred while accessing the SQL Server database or the Office SharePoint Server Search service. If this is the first time you have seen this message, try again later. If this problem persists, contact your administrator.”

Notice in the below picture that the schedule fields are blank.

When accessing “Configure Profile Import”, you will see the following error:

An error has occurred while accessing the SQL Server database or the Office SharePoint Server Search service. If this is the first time you have seen this message, try again later. If this problem persists, contact your administrator.

Also, within the Configure Profile Import page, you will see the following error within Incremental Import Schedule:

“Unable to obtain schedule information. Please verify that the job server is up and connected to the farm.”

As a result, profile imports are unable to be scheduled.

Question and Answer

Question: What causes this?

Answer: This can happen when one of the hidden SSP timer jobs is deleted via stsadm –o deletessptimerjobs. Specifically in this case, the User Profile Incremental Import Job timer job was missing.

Question: How do I validate I’m missing one of the SSP timer jobs?

Answer:   You can run the following from the bin directory:

stsadm –o enumssptimerjobs –title “nameofssp”

SSP Timer Job Id="d29a1e4b-268d-4bb8-90b9-003935407f6b" Display Name="User Profile Change Job"

SSP Timer Job Id="80395702-10aa-46bb-91f8-014bedaf7184" Display Name="Audience Compilation Job"

SSP Timer Job Id="3565824b-d79a-4b5e-9af9-06653c8564b0" Display Name="User Profile Full Import Job"

SSP Timer Job Id="a7e410e3-abdf-4833-b9fe-a2e8b2863182" Display Name="Distribution List Import Job"

SSP Timer Job Id="1fad8b5e-6b2b-48f3-ae79-aa17afb5ca11" Display Name="User Profile Change Cleanup Job"

SSP Timer Job Id="1zfl8b5a-9g3g-59q9-wr41-mm22afb7kia1" Display Name="User Profile Incremental Import Job"

In this particular case, the User Profile Incremental Import Job was missing. Unfortunately, the only supported resolution is creating a new SSP or restoring the SSP from backup. I’ve seen some external public documentation suggesting a fix which includes directly editing the associated SharePoint SSP database. Please be aware that any direct edits to the database is strictly unsupported by Microsoft. Also, I would shy away from using the deletessptimerjob operation for this exact reason.

References:

http://technet.microsoft.com/en-us/library/cc512098(office.12).aspx

http://technet.microsoft.com/en-us/library/cc262978(office.12).aspx

 Thanks,

-Russmax

Introduction into Problem

I had an interesting issue with People Picker performance and SharePoint 2007 that is deserving of some additional documentation.   As many of you know, the people picker is built in feature of SharePoint and assists in looking up users to perform various tasks like adding to AD and/or SharePoint security groups etc…   The people picker also has a control that is used in custom InfoPath forms for use in SharePoint. 

In this particular problem, InfoPath forms contained one or more people picker fields in a Form Library on SharePoint 2007.  

In this case, Participation Not Required is a SharePoint security group.

At various times, attempting to open these InfoPath forms from a forms library within SharePoint would cause an excessive delay, (30 seconds to 2 minutes).  Some dump analysis revealed we were waiting on this function to complete:

System.Security.Principal.NTAccount.TranslateToSids(System.Security.Principal.IdentityReferenceCollection, Boolean ByRef)

The object being passed was the group specified within people picker field on the InfoPath form.  In this particular case, the group was a SharePoint security group.

Cause

The problem is that we will always go to Active Directory first by default to do the lookup via the TranslateToSids function.  This isn’t desirable in this specific case because it’s a SharePoint security group that’s being passed.  The second part of the problem is with the format of the name specified on the people picker field.  The fact that there is no \ in the name causes Active Directory to submit the lookup against every domain in the forest.  In a large Forest, this can become a taxing operation.  Finally, the lookup is resolved within SharePoint.

Work Around

The good news is that we exposed a property that controls how this validation takes place when no \ is present in the name.  Instead of first going to AD, we will perform the lookup within SharePoint first.  This is desirable in certain scenarios such as this one.  Names containing domain\username will still be processed by Active Directory first so no affect their.  The property is the following:

ActiveDirectoryRestrictIsolatedNameLevel

http://msdn.microsoft.com/en-us/library/microsoft.sharepoint.administration.sppeoplepickersettings.activedirectoryrestrictisolatednamelevel.aspx

We exposed this property in the October CU so the SharePoint 2007 farm must be at build 12.0.6520.5000

Related KB’s:

http://support.microsoft.com/kb/976396

http://support.microsoft.com/kb/975002/

 This property is set on the Web Application level and is false by default.   In order to work around the problem, set this property to true.   A few ways for doing this:

PowerShell:

$web=get-web “specifytheurlofthewebapplication”
$web.Site.WebApplication.PeoplePickerSettings
$ps=$web.Site.WebApplication.PeoplePickerSettings
$wa=$web.Site.WebApplication
$ps.ActiveDirectoryRestrictIsolatedNameLevel=$true;
$wa.Update();

C#:

using System;
using System.Collections.Generic;
using System.Text;
using Microsoft.SharePoint.Administration;
using Microsoft.SharePoint;

namespace PeoplePick
{
    class Program
    {
       static void Main(string[] args)
        {

            SPSite oSite = new SPSite("http://moss");

            SPWebApplication myweb = oSite.WebApplication;
                SPPeoplePickerSettings ppSettings = myweb.PeoplePickerSettings;
                ppSettings.ActiveDirectoryRestrictIsolatedNameLevel = true;
                myweb.Update();
                Console.WriteLine("The process has completed successfully");
        }

-Russmax

I moved back to my original role as a Support Escalation Engineer so future posts are going to be random. Some post involve troubleshooting both SharePoint 2007/2010. I’ll also still continue to post the How To/Conceptual stuff with SharePoint 2010.

This post involves an interesting issue I recently got where the customer was unable to complete the Export phase of a Content Deployment job due to the following error:

ServerA to the destination server ServerB, We receiving and error message as "6/1/2010 9:00 AM The system cannot find the file specified. (Exception from HRESULT: 0x80070002) at Microsoft.SharePoint.Library.SPRequestInternalClass.GetFileAsByteArray(String bstrUrl, String bstrWebRelativeUrl, Boolean bHonorLevel, Byte iLevel, OpenBinaryFlags grfob) at Microsoft.SharePoint.Library.SPRequest.GetFileAsByteArray(String bstrUrl, String bstrWebRelativeUrl, Boolean bHonorLevel, Byte iLevel, OpenBinaryFlags grfob) at Microsoft.SharePoint.Deployment.FileSerializer.SaveFile(SerializationInfo info, ExportObjectManager objectManager, ExportDataFileManager fileManager, SPExportSettings settings, SPWeb parentWeb, Boolean isGhosted, String setupPath, String setupPathUser, Byte setupPathVersion, String webRelativeFileUrl, Int32 size, Byte level) _catalogs/masterpage/mycustommasterpage.master

Note: The end of the call stack is helpful here with these errors.

Cause:

In my case, the customer was missing a custom feature which deployed this particular master page.

Validation Techniques:

1. The master page is no longer listed as an item in the associated pages library

2. The feature was missing from the following directory:

C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\Template\Features

3. Running stsadm –o preupgradecheck provided a list of features that were reported as missing

Note: Must have SP2

Question: So how is Content Deployment discovering that the feature is missing?

Answer: The feature is being referenced in the features table. One way to validate is to run the following SQL query against the associated content database:

select * from features with (NOLOCK) where Properties like ‘%nameofcustomfeature%’

Resolution:

The resolution depends on whether or not the SharePoint administrator wants the custom feature.

If the custom feature is wanted:

    1. Copy the feature to the following directory:

    C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\Template\Features

    2. Stsadm –o installfeature –filename customfeaturename\feature.xml

    3. Stsadm –o activatefeature –name customfeaturename –url “http:\\contoso”

If the custom feature is not wanted:

Run the following: stsadm –o deactivatefeature –filename “customfeaturename\feature.xml” –url http:\\contoso”

Note: If this errors stating that the feature is still present, you need to ensure that the associated object is not present in the objects table within the Configuration database.

Run stsadm –o deleteconfigurationobject –id “FeatureID”

Note: The feature ID is the one identified from either of the following steps:

1. The output of the preupgrade check
2. The output of the following: select * from features with (NOLOCK) where Properties like ‘%nameofcustomfeature%’ “Feature ID column”

In my particular instance, the stsadm –o deactivatefeature didn’t purge the references of the feature from the feature table even though it reported as successful. In this case, you must redeploy the feature and then properly deactivate and uninstall.

-Russ Maxwell, MSFT

This blog is intended to fill some gaps and provide a foundation to understand components in the claims model and how these components work together. Claims will provide a huge benefit which I will outline some of those benefits below. I suspect this will turn into a multi blog series so stay tuned for further blogs on this subject. The goal is providing a series of blogs starting from broad and getting more narrow in scope. As the scope is narrowed, a deeper technical progression will take place. SharePoint 2010 has a new approach to authentication\authorization. Instead of using the classic (Integrated) authentication method, it's possible to authenticate and authorize users against external Identity Providers. No longer, are we limited to directory repository's like Active Directory. In fact, it's possible to create custom identity providers and SharePoint will trust and leverage that Identity Provider thus granting external user access to a SharePoint site/document etc...

Special thanks goes to Venky for the knowledge transfer  :)

Claims

An identity provider makes claims about a user. A good example of an identity provider is Live ID. So Live ID will claim to have attributes and their values. For Example:

Identity Provider "provider of the attributes" contains username attribute containing DanCan. A custom identity provider created by a hacker also contains an account with username attribute named DanCan. Both identity providers are making claims about a user. The consumer "SharePoint 2010" must choose which claim it's going to trust. SharePoint 2010 by itself will never trust either claim without being told to do so. In order for SharePoint to use a claim, it must first trust that claim which is setup by you the SharePoint administrator. If claims are trusted, then SharePoint can authenticate and authorize over that claim.

STS

STS is built on Geneva framework which is now called Windows Identity Foundation. The STS (Security Token Service) core responsibility is issuing, managing, and validating security tokens. An STS resides on both an identity provider and SharePoint. STS is built on top of the shared services framework which is why it's listed as a service application within Central Administrator\Manage Service Applications page:

Above, STS is composed of a web service and runs on every SharePoint server.

Authentication

The authentication type is setup at the Web Application level when creating a new SharePoint web application. It's possible to choose either classic authentication or Claims authentication. Each one is discussed below:

Classic

Active Directory authenticates a user, provides an NT Token. The token is used to authenticate to SharePoint. SharePoint consumes that token and it's converted into an SPUser object for authorization.

Note: Authorization is the process of determining what level of access an authenticated user has to a secured resource such as a Site, Document library etc.. The authorization mechanism hasn't changed in SharePoint 2010 and we ultimately still use an SPUser object to authorize.

Claims

After a trust is established between SharePoint and an Identity provider, web applications can be set with Claims authentication type instead of classic. If a client attempts to authenticate to a claims aware web application, SharePoint redirects a client to the associated trusted identity provider. The identity provider authenticates clients and provides a security token. That token could be either of the following:

· NT Token

· SAML Token

This security token is this passed to SharePoint STS. In short, the STS will validate the token "Claims Based Identity" and generate a new security "SAML" token back to the client. This token is generated by SharePoint and for SharePoint. The client sends this SAML token to SharePoint to prove that he/she is officially authenticated. SharePoint validates and authenticates user and an SPUser object is created and is used for authorization.

Steps for Claims Sign-In:

1. Client hit SharePoint site via HTTP (Get)
2. SharePoint redirects client to Identity Provider in order to get a security token
3. Client attempts to authenticate to trusted Identity Provider
4. The identity provider's (Security Token Service) will validate the username and password and provide a security token to a client.

Note: A security token could be a Windows NT Token, SAML token, or FBA token

5. The client has a security token (authenticated) and submits it to SharePoint STS "Security Token Service"
6. SharePoint STS receives security token from client and determines if we trust the issuer of that token "Identity Provider"
7. STS then performs claims augmentation
8. STS issues client new SAML token
9. Client request resource "site" with new SAML token
10. SharePoint consumes SAML token, "validates authentication successful", and builds an SPUser object in order to authorize to the secured resource

Mixed Authentication

In SharePoint 2007, to use additional authentication provider, you had to extend the web application and drop it in a different zone so it would contain a different URL. SharePoint 2007 wasn't flexible in terms of specifying multiple authentication types in a single un-extended web application.

Multi Authentication

In SharePoint 2010, it's possible to configure multiple authentication types for a single web application. This provides 2 benefits:

1. No longer required to extend web-application for the purpose of adding additional authentication types

2. Can have a single web application use multiple authentication types which provides the ability to serve a single URL!

Note: You can still extend web-applications and assign one or more authentication types to it if a business justification calls for that.

FBA

FBA users no longer uses an ASP.Net identity. FBA is now claims aware and the SharePoint STS facilitates the authentication process. Once user is authenticated, the SharePoint STS provides a SAML token to the client.

Note: When creating a web application designated for FBA, you must specify claims authentication type.

STS (federated equivalent of a domain controller) "issues tokens"

Basic FBA Sign-in process:

1. User signs in via FBA with credentials
2. SharePoint STS calls membership provider to authenticate
3. SharePoint STS calls role provider to get all the roles for the user
4. Post successful authentication, a SAML token is generated by the SharePoint STS and passed back to the user
5. The user then authenticates to SharePoint with SAML token and authentication is officially completed

For setup steps, please see my blog for more details.

How Claims works with Services

Accessing Internal Services

Within a Single Farm:

The classic example is a user performing a search. The WFE's (Server1) search web part talks to service application proxy. The associated search service application proxy calls the local STS to get a SAML token for the user. Once SAML token is collected, the search service application proxy then calls a server running the Query Processor via WCF call. I'll call this server, "Server 2". Server 2 receives the incoming request and validates the SAML token against its local STS. Once validated, Server 2 connects to various components to gather, merge, and security trims search results. Server 2 sends the trimmed search results back to Server 1 which are then presented to the user.

Accessing External Services

SharePoint 2010 STS can manipulate a SAML token in order to present it to an external web service. The way it presents the identity depends on the type of external web service. The goal is preventing the additional prompt for credentials so that a full Single Sign-On (SSO) experience is possible. The STS is comprised of the WIF "Windows Identity Framework" and also the C2WTS. Each component is used dependent upon the type of external service accessed.

C2WTS = Claims to Windows Token Service

If accessing a native windows application that expects a Kerberos ticket. Within SharePoint STS, we use C2WTS to use existing SAML token in order to create a windows token (Kerberos ticket) to authenticate.

http://msdn.microsoft.com/en-us/library/ee517278.aspx

SharePoint STS

Can be used to just issue SAML token to pass to external systems that support SAML tokens

Secure Store Service

SharePoint can be used to connect to a legacy LOB systems which requires credentials. (SSS) Captures credentials and uses them on web service call to login and go inside.

http://msdn.microsoft.com/en-us/library/ee557754.aspx

Thanks,

Russ Maxwell, MSFT

My beta rotation ended this May and I’m back in my role of supporting Premier Customer’s with various SharePoint issues.   Please no v2 issues  :)     This was my first experience joining any sort of beta program at MSFT.  I really feel fortunate being involved in this particular beta because we have made so many improvements in this build of SharePoint.  I’ll continue to blog on various SharePoint topics but will probably introduce more troubleshooting type blogs since our documentation on SharePoint 2010 is great. 

Special thanks to Sheyi, Dan W., Doron, Luca, Jim, Radu and countless others I’ve interacted with over the past year…

I thought I would post some links to some SharePoint 2010 and SharePoint 2007 content that I’ve found along the way.. 

SharePoint 2010 Developer Training

Microsoft SharePoint Product Group Blog

Microsoft Enterprise Search Blog

SharePoint FaceBook 

SharePoint Updates on FaceBook

TechNet

TechNet Library

Microsoft SharePoint Designer Team Blog

Detailed list of SharePoint 2007 Cumulative Updates

Detailed list of SharePoint 2010 Cumulative Updates

Stay tuned…  More blog content is coming soon  :)

-Russ Maxwell, MSFT

This is a short blog which briefly discusses Shared Service Architecture in a multi farm environment.   I will discuss some core components and describe how these components work together.  I will not go into exact setup steps due to the fact TechNet has a great run through here:

Exchanging Certs:      http://technet.microsoft.com/en-us/library/ee704552(office.14).aspx

Publishing a Service Application:    http://technet.microsoft.com/en-us/library/ee704545(office.14).aspx

Connecting to a Published Service Application:  http://technet.microsoft.com/en-us/library/ee704558(office.14).aspx

Special shout-out goes to Ram and Sheyi for some great contributions to this blog.   I recommend reviewing Shared Service Architecture Part 1 blog to build a foundation before continuing further.

Basics

SharePoint 2010 Shared Service architecture has been revamped to provide great flexibility in a single farm.   This flexibility in a single farm has been extended to include multiple farms.   For Example, Shared Services can exist in a farm I’ll call “Services Farm”.   All content like SharePoint sites and My-Sites reside in a farm called “Consumer Farm”.   A services farm consists of multiple shared services that can be published and made available to remote farms.   The Consumer Farm can connect to a published service application from the services farm and consume from it.  

For Example:

Services Farm: Published Search Service Application and give Admin access via permissions button for the specified service application.

Consumer Farm:  Consumes Search Service Application from Services Farm

Action from Consumer Farm:  User accessing a site performs a search

Results from Services Farm: Search results are pulled from Services Farm.

Initial Deployment is a four step process. 

1. Setup trust between farms by copying\importing  root\STS security token certificates

          Note:  The STS cert is provided to the Publishing farm from the Consumer farm

    2.   On the services farm, publish shared service applications making them available to other farms

    3.   Set Permissions on who can access the published shared service applications

    4.  The consumer farm will use Connect button to discover the services in a destination farm and to create a Proxy to a Shared service in a remote farm.

Note:  A successful attempt at step 4 creates a service application proxy and places in a proxy group within the Consumer farm.  Only web applications mapped to the same proxy group will consume from this published service application.

Topology Discovery Service Application

This is known as the Application Discover and Load Balancer Service Application, “I’ll refer to as the topology service”. Within IIS, it’s listed as Topology and runs using WCF web service. 

Central Administrator:

IIS:

Without a Topology Discovery Service Application, consuming services across farms is not possible.  Why it’s not possible will be answered when the entire blog has been reviewed.

Functions of Topology Service on publishing farm

In a services “publishing” farm, the topology service provides a list of published shared service applications that are available for consumption.  It also keeps track of which Published Shared Service application instances are online\offline so that it consistently maintains a fresh list of URI’s.   It maintains a cache of these URI’s to avoid multiple round trips to the configuration database.

Functions of Topology Service on consuming farm

The topology service functions in a consuming farm take effect only after a connection is established.  That is, a service application proxy is created on a consumer farm that connects to service application in publishing farm.   In a consumer farm, the topology service connects to the Publishing farms (remote) topology service to retrieve a list of added/removed URI’s.  A timer job controls when this takes place called the Application address refresh timer job.  The default is every 15 minutes.  When it runs, it calls the Publishing farm’s Topology Web Service to retrieve the updated service end-point URI list.  If there is any change, it will update the associated service application proxy.

Load Balancer 

The load balancer component provides two functions in a content farm:

1. It ensures request from service application proxies are evenly distributed to remote published shared service instances.

2. Maintains a list of fresh URI’s

On a consumer farm, when a service application proxy is first provisioned that is consuming service in a remote publishing farm,  the service application proxy directly connects to the publishing farms remote topology service to get the list of available URI’s (See the troubleshooting section for more details).  It stores these URI’s in a load balancer cache.  Therefore, each service application proxy in a consuming farm contains a unique load balancer component.   After the initial provision, the load balancer component will not attempt to connect to the remote topology service again to receive URI updates.   It will only receive updates from the local topology service via the Application address refresh timer job.  

Question: What if a load balancer discovers service connection endpoints are down?

Answer: It maintains freshness in that when it discovers service endpoints are no longer available it marks them as down for a period of time and will not use it until that time period has expired.   For further service calls, the load balancer tries to get the next available URI from its cache.   When the time period is over, it will bring the downed link back online.

Troubleshooting Topology Service (basics)

This section will uncover what takes place behind the scenes “what the trace logs look like” on various aspects of multi farm communication.   I’ll cover each action in its own section.  The data provided will display a glimpse of healthy activity and can be used as a reference based on the type of action taken.  

Connecting to Topology Service to consume Search Service Application

In this example, SharePoint administrator in consumer farm is connecting to topology service in publishing farm and selecting a Search Service Application to consume from it: 

03/08/2010 09:03:03.08         w3wp.exe (0x0A00)               0x07AC    SharePoint Foundation           Logging Correlation Data       xmnv       Medium                Name=Request (POST:http://consumerfarm:4444/_admin/ServiceApplicationConnect.aspx?IsDlg=1)           bcb6e7a2-4749-4c2d-99d0-49ad45544e1d

03/08/2010 09:03:03.11         w3wp.exe (0x0A00)               0x07AC    SharePoint Foundation           Topology 84cy        Verbose   Retrieving shared service application information for url: 'https://servicesfarm:32844/Topology/topology.svc'          bcb6e7a2-4749-4c2d-99d0-49ad45544e1d

03/08/2010 09:03:03.11         w3wp.exe (0x0A00)               0x07AC    SharePoint Foundation           Topology 84d0        Verbose   The specified url is not topology service load balanced. Assuming the uri is hardware load balanced for the topology service 'https://servicesfarm:32844/Topology/topology.svc'          bcb6e7a2-4749-4c2d-99d0-49ad45544e1d

03/08/2010 09:03:03.30         w3wp.exe (0x0A00)               0x07AC    SharePoint Foundation           Topology e5mc       Medium  WcfSendRequest: RemoteAddress: 'https://servicesfarm:32844/Topology/topology.svc' Channel: 'Microsoft.SharePoint.ITopologyWebServiceApplication' Action: 'http://tempuri.org/ITopologyWebServiceApplication/EnumerateSharedServiceApplications' MessageId: 'urn:uuid:ace71aeb-f378-41d2-8593-e43af416c98b'        bcb6e7a2-4749-4c2d-99d0-49ad45544e1d

03/08/2010 09:03:03.44         w3wp.exe (0x0A00)               0x07AC    SharePoint Foundation           Logging Correlation Data       xmnv       Medium                Name=Request (GET:http://consumerfarm:4444/_admin/SelectApplication.aspx?proxyId=&typeId=&IsDlg=1)            1aeaeebc-e766-41af-89c7-9d4ece0f24b0

03/08/2010 09:03:03.44         w3wp.exe (0x0A00)               0x07AC    SharePoint Foundation           Topology 84d3        Verbose   The specified url is a topology service url. Not filtering retrieved results              bcb6e7a2-4749-4c2d-99d0-49ad45544e1d

03/08/2010 09:03:03.45         w3wp.exe (0x0A00)               0x07AC    SharePoint Foundation           General   6t8b         Verbose   Looking up context  site http://consumerfarm:4444/_admin/SelectApplication.aspx?proxyId=&typeId=&IsDlg=1 in the farm SharePoint_Config2             140e4dd4-fa79-47f9-b0c5-9e638dad9258

03/08/2010 09:03:03.45         w3wp.exe (0x0A00)               0x07AC    SharePoint Foundation           General   6t8h         Verbose   Found typical site / (cd40b0f0-3b69-484a-94b6-30bdb2def735) in web application SPAdministrationWebApplication.  140e4dd4-fa79-47f9-b0c5-9e638dad9258

03/08/2010 09:03:03.50         w3wp.exe (0x0A00)               0x07AC    SharePoint Foundation           Topology 84d4        Verbose   Getting supporting proxy types for application 'Search Service Application 1' using proxyId '00000000-0000-0000-0000-000000000000' and typeId '00000000-0000-0000-0000-000000000000'       140e4dd4-fa79-47f9-b0c5-9e638dad9258

03/08/2010 09:03:03.50         w3wp.exe (0x0A00)               0x07AC    SharePoint Foundation           Topology 84d5        Verbose   Found supporting proxy '' for application 'Search Service Application 1'    140e4dd4-fa79-47f9-b0c5-9e638dad9258

03/08/2010 09:03:11.67         OWSTIMER.EXE (0x0FEC)      0x11AC    SharePoint Server Search       Administration        dk3p        Verbose SearchService.Provision: Begin.             5483fb53-a1f1-4cec-b08f-803a4b15d422

03/08/2010 09:03:11.67         OWSTIMER.EXE (0x0FEC)      0x11AC    SharePoint Server Search       Administration        dk3s        Verbose SearchService.Provision: End.                5483fb53-a1f1-4cec-b08f-803a4b15d422

Communication to web services

In this example, user Dan is attempting to edit his profile and populate some properties.   He is logged into Consumer farm which is consuming Profile Service application in services farm.  To find which URI is being used by the User Profile Service Application Proxy, filter the ULS log to (category = Topology and Message contains = WcfSendRequest). 

03/08/2010 06:05:01.23         OWSTIMER.EXE (0x0FEC)      0x1134    SharePoint Foundation           Topology e5mc       Medium  WcfSendRequest: RemoteAddress: 'http://servicesfarm:32843/fe60ff7b10af4828a2b76ca61ee38da6/ProfilePropertyService.svc' Channel: 'Microsoft.Office.Server.UserProfiles.IProfilePropertyService' Action: 'http://Microsoft.Office.Server.UserProfiles/GetProfileProperties' MessageId: 'urn:uuid:6868c5bd-a277-4593-ba77-4821840c713f'           

03/08/2010 06:30:07.16         OWSTIMER.EXE (0x0FEC)      0x0A60    SharePoint Foundation           Topology e5mc       Medium  WcfSendRequest: RemoteAddress: 'http://servicesfarm:32843/fe60ff7b10af4828a2b76ca61ee38da6/ProfileDBCacheService.svc' Channel: 'Microsoft.Office.Server.UserProfiles.IProfileDBCacheService' Action: 'http://Microsoft.Office.Server.UserProfiles/GetUserData' MessageId: 'urn:uuid:cd65949f-cccd-425c-8c43-de26e4db9c79'             ffbe7cb1-0db1-4966-8196-f533c54eaccf

This is ideal for troubleshooting in that you can see the URI specified.  Also, this is a valid first step to validate were successfully reaching the service application instance.   This could also be coupled with Netmon and\or Fiddler to get a complete picture.

Application address refresh timer job

Again, the Application Address refresh timer job executes on consumer farm and connects to remote Services farm’s topology service.  It fetches changes (Added\Removed URI’s).   In the following example, the consumer topology component is fetching the latest changes for one published service application.   In this example, no updated changes are discovered.

03/08/2010 14:32:16.48         OWSTIMER.EXE (0x0FEC)      0x1200    SharePoint Foundation           Logging Correlation Data       xmnv       Medium                Name=Timer Job job-spconnectedserviceapplication-addressesrefresh             5af1cadb-02f8-4522-a6ab-a5451a738e26

03/08/2010 14:32:15.47         OWSTIMER.EXE (0x0FEC)      0x0AD4   SharePoint Foundation           Timer      5utp         Verbose   Scheduled timer job Application Addresses Refresh Job, id {879484E3-BFEB-436E-9FC1-B26C810BCA29} at 08 Mar 2010 14:32:16 -0800 (now is 08 Mar 2010 14:32:15 -0800)               

03/08/2010 14:32:16.48         OWSTIMER.EXE (0x0FEC)      0x1200    SharePoint Foundation           Topology e5mc       Medium  WcfSendRequest: RemoteAddress: 'https://servicesfarm:32844/Topology/topology.svc' Channel: 'Microsoft.SharePoint.ITopologyWebServiceApplication' Action: 'http://tempuri.org/ITopologyWebServiceApplication/GetEndPoints' MessageId: 'urn:uuid:282a3a03-4a87-4d81-9fb0-be529953cdf6'       5af1cadb-02f8-4522-a6ab-a5451a738e26

03/08/2010 14:32:16.48         OWSTIMER.EXE (0x0FEC)      0x1200    SharePoint Foundation           Monitoring              nasq        Verbose   Entering monitored scope (ExecuteWcfOperation:http://tempuri.org/ITopologyWebServiceApplication/GetEndPoints)              5af1cadb-02f8-4522-a6ab-a5451a738e26

03/08/2010 14:32:16.53         OWSTIMER.EXE (0x0FEC)      0x1200    SharePoint Foundation           Topology 3ls4         Verbose   Application addresses for connected application: 46ad8378-690c-4568-b8d8-2485b1b89c1b are up-to-date             5af1cadb-02f8-4522-a6ab-a5451a738e26

03/08/2010 14:32:16.53         OWSTIMER.EXE (0x0FEC)      0x1200    SharePoint Foundation           Monitoring              b4ly         Verbose   Leaving Monitored Scope (ExecuteWcfOperation:http://tempuri.org/ITopologyWebServiceApplication/GetEndPoints). Execution Time=49.8323110898173                5af1cadb-02f8-4522-a6ab-a5451a738e26

03/08/2010 14:32:17.31         OWSTIMER.EXE (0x0FEC)      0x1200    SharePoint Foundation           Monitoring              b4ly         Medium  Leaving Monitored Scope (Timer Job job-spconnectedserviceapplication-addressesrefresh). Execution Time=830.343800678578             5af1cadb-02f8-4522-a6ab-a5451a738e26

-Russ Maxwell, MSFT

Several things have changed in SharePoint 2010 Query.   Query infrastructure is also componentized so now you only provision what you need.   This blog will go through will define the query components, how they work together, and how to provision them. 

Special shout-out goes to Jon Waite for his valuable technical input\review…

Query Basics

Just like crawl, Query has been componentized as well and the following goals are met:

• Sub-second query latency
• Index is no longer a single point of failure and is stored on Query servers
• Query consists of components which can be scaled out among multiple servers to improve performance

Query Flow

1. A search is performed by a user
2. The WFE serving the call uses the associated search service application proxy to connect to a server running the Query and Site Settings Service also known as the Query Processor.  It uses WCF for this communication.
3. The QP will connect to the following components to gather results merges\security trims and return results back to WFE:

• Query Component - holds entire index or partition of an index
• Property Store DB – holds metadata\properties of indexed content
• Search Admin DB – holds Security Descriptors\Configuration data

     4.   WFE displays search results to the user

Several Query components can be scaled out as an index\property store grows.  A single search service application can have multiples of the following:

• Property Store DB
• Query Components
• Query Processors

Query Component and Property Store DB

I’ll use Query server and Query Component interchangeably throughout the blog.  A Query Server is a server that runs one or more Query Components.   These servers hold a full or partial of the search index.  Query Servers are now the sole owner of storing the index on the file system.   As stated from previous post, the indexer crawls content and builds a temporary index.  The Indexer propagates portions of the temporary index over to Query Server to be indexed.  Query Servers contain a copy of the entire or partial index referred to as an Index Partition.  Query components run under the context of an Index partition.   Query components are responsible for serving search queries.  Query component runs under MSSearch.exe.   A query component is mapped to only one Property Store DB.  By now, you should’ve noticed that we split up the databases (For Example: Property Store DB and Crawl DB).   By separating these databases the following has been accomplished:

• Overall Database size is reduced
• Database performance is improved

Also, by carving out the databases, performance hits like writing crawled data to Crawl Store DB won’t affect tasks like serving Queries “query performance” which heavily depends on the Property Store DB. 

It’s possible to provision multiple Property Store databases and Query components for a single Search service application.  The reasons for doing this are plentiful and most of the reasoning will be explained throughout this post.   Query components can be provisioned to partition an index and\or mirror an index in order to provide fault tolerance.  Both of these components can be created by using either Central Administrator or PowerShell.  To simplify things a bit I’ll cover how to do it in Central Administrator.  In order to make changes to the Search topology, you must access the Search Administration page via the following:

Central Administrator\Application Management\Manage Service Applications\Select Search Service Application and select Manage from Ribbon

Scroll to the bottom of the page and this is where you can view\change the search topology. 

Provisioning happens in 3 stages:

1. Hit Modify button
2. Select New Property Database or Query Component and enter appropriate options at your discretion
3. Apply Topology Changes

Fault tolerance + Performance

Query Component (Fault tolerance)

It’s highly recommended to create fault tolerance with your index.   This is accomplished by mirroring a Query component assigned to a different server.   Under the Search Application Topology, you can simply select the Query Component and Add mirror:

The end result is a second query component within the same Index Partition.

Note:  The Query Processor will distribute requests across both Query Components. 

Question: I don’t want Queries being served by one of my mirrored Query Components.

Answer:  On the Add mirror query component page, you can check the following option:

This doesn’t eliminate the failover query component from receiving queries.  The Query Processor will prefer Query Components not marked as fail over (active).  If all active Query Components are down, then Query Processor will submit requests to Query Components flagged as fail over. 

Property Store (Fault tolerance)

We fully support SQL mirroring to achieve fault tolerance with Property Store DB’s on the backend.

Query Component (Performance)

In previous builds of SharePoint, every query server stored the entire index.   While this achieved fault tolerance it didn’t help with performance.    There is a direct correlation between the size of an index and query latency.  The size of an index can easily become a bottleneck for query performance.  

For Example:

• Index contains 10 million documents =  Average of 2 seconds per query
• Index contains 20 million documents = Average of 4 seconds per query

This problem has been solved in SharePoint 2010.   Index partition can contain the entire index or a portion of the index.  By creating additional query components, a new index partition is created and owns a portion of the index. 

For Example:

If the entire index is 8 GB and contains 20 million documents:

Holds 50%: 4GB of index\10 million documents:  Query Server 1 - Index Partition 1   

Holds 50%: 4GB of index\10 million documents:  Query Server 2 - Index Partition 2

By partitioning large indexes, query times are reduced and a solution to this type of bottleneck can be solved.   Partitioning an index is as simple as provisioning new Query Components from the Search Application Topology section in Central Administrator.

For Example:

Question: If an index is partitioned out with multiple Query Components, how does the crawler distribute the indexed content?

Answer: The crawler evenly distributes crawled content to Index Partitions using a hash algorithm based on Doc ID’s.  

Property Store DB (Performance)

Just like Query components, Property Store DB can be scaled out and share the load of the metadata stored in the Property Store DB.   If the Property Store DB becomes a bottleneck due to the size of the database and\or strains the disk subsystem with high I/O latency on the back end, a new Property Store DB can be provisioned to share the load.  Just like the Crawl DB, the Property Store DB is useless unless it’s mapped to something.  In this case, a Property Store DB must be mapped to a Query component.   If a decision is made to provision an additional Property Store DB to boost performance, an additional non-mirrored Query Component must be provisioned and mapped to it. 

The following is a true statement:

“Creating an additional Property Store DB requires the Index to be partitioned off because provisioning a new Query Component is required”.     

Query Processor

Great, so understanding Property Store DB and Query component scale out is only half of the battle.   The Query Processor remains and still plays a vital role in Search 2010.  The Query processor is responsible for processing a Query and runs under w3wp.exe process.  It retrieves results from Property Store DB and the Index\Query Components.   Once results are retrieved, they are packaged\security trimmed and delivered back to the requester which is the WFE that initiated the request.  The Query Processor will load balance request if more than one Query Component (mirrored) exists within the same Index Partition.  The exception to this rule is if one of the Query Component’s is marked as fail over only. 

Question: What if I partitioned off my index and I have multiple Query Components provisioned each serving a partition of the index?  How does Query Processor know which partition to connect to in order to accurately retrieve results?

Answer:  It doesn’t!  The Query Processor will connect to every single non-mirrored Query component that contains a partition of the Index to retrieve results.  

Question:  What if I created multiple Property Store Databases for performance reasons?   How does Query Processor know which Property store to connect to in order to accurately retrieve results?

Answer: It doesn’t!  The Query Processor will connect to every single Property Store DB to retrieve results.  

In SharePoint 2007, the Query Processor ran on any WFE.   In SharePoint 2010, any server can run the Query Processor.  It’s no longer tied into a server running the Query role.   You provision Query Processor role on a server by performing the following steps:

1. Within Central Administrator, System Settings, Service on Server
2. Start the Search Query and Site Settings Service

Note:  Post provision a new web service is created within IIS on that server.

Query Processor Scale Out

Just like the Query Component and Property Store DB, the Query Processor role can be scaled out to multiple servers.  If the Query Processor is a bottleneck, For Example:

· Not able to keep up with inbound requests or perhaps the box and/or associated W3WP.exe process hosting Query Processor is CPU\Memory bound.

In this case, you provision additional Query Processors as needed.  By provisioning additional query processors, requests will be load balanced in a round robin fashion to each server hosting a Query Processor.    

The same case can be made for achieving fault tolerance.  By having two servers hosting Query Processor role, if one goes down, the other will be used. 

Query Processor functions in Parent\Child Farm

In a Publishing/Consumer farm scenario, the Query Processor always runs in the farm where the Search Service Application resides.   So if Search Service Application resides in Publishing farm, Query Processor only runs in publishing farm.   The Consumer farm utilizes the associated Search Service Application proxy to make the connection over WCF to a Query Processor in the publishing farm.

Observe is Step 1 and Taking Action is Step 2

Before arbitrarily provisioning new query components and property store DB’s, observe the current environment\query health so some evidence can be gathered before making this important decision.  The obvious reasons of Fault Tolerance and Query Latency are covered in the previous sections so I won’t discuss those further.  Observing for System\Hardware bottlenecks is a good first step before considering adding more Query Components\Property Store DB’s. 

Monitoring Query Server

Observation:  The Query server is almost maxed on CPU and\or is at the peak of available physical memory and query latency has increased as a result.

Action Taken:  Provision a new query component 

Monitoring SQL Server

Observation:  Property Store DB is I/O bound on SQL and disk latency is unexpectedly high.

Action Taken:  Provision a new Property Store on same/different SQL server

Important:  These are very basic methods on approaching system bottlenecks.   For Example, don’t assume from a general observation of a spiked CPU would automatically require provisioning additional query components.   More analysis would be required.  Such as finding answers to the following questions:

1. Does CPU only spike during crawl times?
2. Which process is spiking?
3. As the overall size of the index/Property Store DB increased?
4. Does SP health monitoring or Performance monitor reveal anything of use?
5. Etc….

Thanks,

Russ Maxwell, MSFT

Several things have changed in 2010 SharePoint search.   Search is now componentized so now you only provision what you need.   This blog series will go through what these components are, how they work together, and how to provision them.  The first blog we will focus on scaling Crawl.  The next blog will be Query and etc...  

Special shout-out goes to Jon Waite for his valuable technical input\review…

Basics

No longer is search tied into an SSP.  Search consists of a Search Service Application, Search Service Application proxy, web services, search service instance, and the following SQL databases by default:

• Search Service Application DB
• Search Service Application Crawl Store DB
• Search Service Application Property Store DB

Search takes advantage of the shared services framework built on SharePoint foundation.   For more details on shared services see my previous blog.  It’s necessary to define the components that make up search.   Each component plays an important role.  This blog will focus on the Crawl Component and the SQL Crawl Database.

It’s possible to provision a search service application\proxy in 3 ways.

1. Within Central Administrator\Manage Service Applications page
2. Within Central Administrator\Farm Configuration Wizard
3. Using PowerShell.   See this blog for more details

Crawl Component and Crawl Database

I’ll use indexer and crawler component interchangeably throughout the blog.  An indexer is simply a physical server that host one or more crawl components.   The huge change in 2010 is the indexer no longer stores a copy of an index.  As items are being indexed, they are streamed\propagated to a Query server.  Because the indexer no longer holds a physical copy of the index,   it’s no longer a single point of failure.  By default, when you provision a search service application using Central Administrator or Farm Configuration Wizard, a Crawler component and Crawl database is provisioned for you.   For lack of better words, a crawl component’s job is to crawl\index content.  The crawl component runs within MSSearch.exe process and the MSSearch.exe process is a windows service “SharePoint Server Search 14”. 

It’s possible to provision multiple Crawl databases and Crawler components for a single Search service application.  The reasons for doing this are plentiful and some of the reasoning will be explained throughout this post.   When a crawler component is provisioned, it requires a mapping to a SQL crawl database.   Both of these can be created by using either Central Administrator or PowerShell.  To simplify things a bit I’ll cover how to do it in Central Administrator.  In order to make changes to the Search topology, you must access the Search Administration page via the following:

Central Administrator\Application Management\Manage Service Applications\Select Search Service Application and select Manage from Ribbon

Scroll to the bottom of the page and this is where you can view\change the search topology.

To provision a new crawl database within Search Admin Page:

1. Select Modify
2. New Crawl Database
3. Populate the new crawl database page and select OK
4. Select Apply Topology Changes

To provision a new crawl component

1. Select Modify
2. New Crawl Component
3. Populate the New Crawl Component  page  and select OK

Note: Server specified will host crawl component.   Associated Crawl Database is where you specify which Crawl   database this crawl component will be mapped to.  In this case, I will map to my newly created one.

    4.    Select Apply Topology Changes

The end result is a secondary crawl database and crawl component.

Fault tolerance + Performance

Just because an index doesn’t physically reside on the indexer doesn’t mean that you should only have one.   For example, if only one crawler component is provisioned and the index server hosting that component fails, a major part of search is broken in that no further crawls will take place.  Fault tolerance can be achieved by provisioning a secondary crawl component on a secondary server.   A crawl component can only map to one SQL Crawl database.  Multiple crawl components can map to the same Crawl database.    By having multiple crawl components mapped to the same crawl database, fault tolerance is achieved.   If the server hosting crawl component 1 crashes, crawl component 2 picks up the additional load while 1 is down.   This achieves fault tolerance for the Indexer but what about fault tolerance for the Crawl DB?   We fully support SQL mirroring and the Crawl DB can be mirrored in SQL to achieve fault tolerance.

Performance is improved in this setup because you effectively now have two indexers crawling the content instead of one.   If you’re not satisfied with crawl times, simply add an additional crawl component mapped to the same crawl DB.   The load is distributed across both index servers.

Question:  If I specify two indexers to crawl the same content “two crawl components” map to the same Crawl DB, is it possible that the indexers might both attempt to crawl newly discovered items?

Answer:  No overlapping would occur. Items are crawled and “picked up” in batches by both index servers for processing.    If Indexer 1 picks up and processes Batch 1, then Index Server 2 will process Batch 2. 

Crawl Distribution

Crawl Distribution can be achieved by provisioning the following:

Crawl Component 1   -->  Crawl DB 1

Crawl Component2  -->  Crawl DB 2

By having multiple crawl components each mapped to a unique Crawl database, each host is assigned to only one Crawl DB at crawl time.  A host is simply an address defined in a content source.    So if I have two web applications provisioned using Host Headers called Sales.Contoso.Com and TPSReports.Contoso.Com, each one is a unique host.   I provision a new Search Service application with the following setup:

Index Server 1 “Crawl Component 1”   <-->  Crawl DB 1

Index Server 2 “Crawl Component2”  <-->  Crawl DB 2

When I start a crawl, each host in this example is evenly distributed so it will look like this. 

Sales.Contoso.Com  -->    Index Server 1 “Crawl Component 1”   <-->  Crawl DB 1

TPSReports.Contoso.Com -->  Index Server 2 “Crawl Component2”  <-->  Crawl DB 2

Indexers “Server hosting a crawl component” associated to that crawl database crawl that host.     When multiple crawl databases exist, an attempt is made to distribute these evenly.   In this example, I only have two hosts.   What if I add a third host after the fact, how does the crawler determine which Crawl DB the host will be assigned to?  The decision is based on the # of items\doc id’s are stored in the Crawl DB.   From the example above, Sales.Contoso.Com has 300,000 items and assigned to Crawl DB 1 and TPSReports.Contoso.Com has 200 items and assigned to Crawl DB 2.  The following hosts are added as content sources:

HR.Contoso.Com  has 2,000 items

Facilities.Contoso.Com has 8,000 items

When a crawl is initiated, both hosts will be assigned to Crawl DB 2 based on the fact Crawl DB 2 contains less Doc ID’s than Crawl DB 1.   Host Distribution isn’t solely determined by the # of Host but rather the # of items in a particular Crawl DB.  By default, a Crawl DB with fewer Doc Id’s will get always get assigned a new host.   It’s possible to control this automated behavior using Host distribution rules.

Note: Having more crawl databases than crawl components makes no since and is a waste of disk space/resources on the SQL server.

Question: One Crawl Component\Crawl Database exists and already has crawled all hosts in the farm.   A decision is made for various reasons to add an additional Crawl Component\Crawl Database.   The SharePoint Administrator would like to forklift half of the host addresses already indexed in original Crawl DB over to the newly created Crawl DB.  

Answer: Once a new Crawl Component\Crawl Database is provisioned, a few ways exist on how to move half of the host:

· Option 1: Reset the Index and perform a Full Crawl.

· Option 2: Add Host Distribution Rules for half of the host and redistribute.  Once complete, host distribution rule\rules can be removed.  

Controlling Host Address Distribution

In some instances, a SharePoint administrator will want greater control on the decision making that goes into assigning host to a specific indexer\crawl DB.  This is controlled through the use of host distribution rules.  This is accessible as a link on the Search Admin page. 

When you add a host and apply, if the host is already distributed to a crawl store, the crawler will be paused and the content will be physically moved and assigned to the crawl DB you select after selecting Redistribute Now button below.  This is a resource intensive operation.

Need more control?

You can also control whether or not a crawl DB will be used for only Host specified by a Host Distribution rule.   When provisioning a new crawl component the last option at the bottom is the following:

By marking a crawl database available to only hosts stored in Host Distribution rules, the indexer is aware of the change and will not automatically allocate newly discovered Host to this crawl database at crawl time.   Only host addresses that are defined in Host Distribution rules mapped to a Crawl DB with this property set will get assigned. 

A good example of when a SharePoint administrator might take advantage of this functionality is when a newly added Host consists of several million items.  This would provide a dedicated Crawl database for that host assuming Host distribution rule was created prior to crawling.  Since the database is dedicated to this host, the database will be exempt from receiving any new host.

Question: Can I combine fault tolerance and shorter crawl times with host that are defined in host distribution rules and are mapped to a crawl database marked as dedicated?

Answer: Sure, it’s as simple as defining another crawl component mapped to the dedicated crawl database.   Remember, multiple crawl components can map to the same crawl DB which effectively achieves fault tolerance + performance increase in shorter crawl times.

Question:  Wait just a minute!  I’m happy with my single crawl component\crawl database.  I’m retiring the Indexer “server hosting crawl component” and I want this new shiny Server to become my indexer.  Do I need to create a new crawl component?

Answer:  Nope, no need to create additional crawl components for migrating to a new index server.   It’s as simple as editing the current crawl component via Search Admin Page\Search Application topology and specifying the new server.  

Observe is Step 1 and Taking Action is Step 2

Before arbitrarily provisioning new crawler components and crawl DB’s, observe the current environment\crawl health so some evidence can be gathered before making this important decision.  The obvious reasons of Fault Tolerance and Decreased crawl times are covered in the previous sections so I won’t discuss those further.  Observing for System\Hardware bottlenecks is a good first step before considering adding more Crawl Components\Crawl DB’s. 

Monitoring Index Server

Observation:  The index server is almost maxed on CPU and\or is at the peak of available physical memory and crawl times have increased as a result.

Action Taken:  Provision a new crawl component on a different server mapped to the same crawl DB

Monitoring SQL Server

Observation:  Crawl Database is I/O bound on SQL and disk latency is unexpectedly high.

Action Taken 1:  Provision a new Crawl Database on same SQL server “on a different set of spindles” and forklift half of the host over to it via Host Distribution Rules.

or

Action Taken 2:   Provision a new Crawl Database on a different SQL server with a more suitable disk subsystem and forklift half of the host over to it via Host Distribution Rules. 

Note: Provisioning a crawl DB requires provisioning a new Crawl Component.

Observation:  SQL Server Memory\CPU is peaking and is unable to sustain the heavy load during crawl times.

Action Taken:  Provision a new SQL server and Crawl DB to reside on the newly created Server.   Forklift half of the host over via Host Distribution Rules.

Important:  These are very basic methods on approaching system bottlenecks.   For Example, don’t assume from a general observation of a spiked CPU would automatically require provisioning additional crawl components.   More analysis would be required.  Such as finding answers to the following questions:

1. Does CPU only spike during crawl times?
2. Which process is spiking?
3. Any host recently added to content sources?
4. Does SP health monitoring or Performance monitor reveal anything of use?
5. Etc….

Thanks,

Russ Maxwell, MSFT

Part 2 - Configuring Hosting​

Configuring hosting requires powershell so the steps are all based off of using it.  The following steps take you through the process of setting up Multi-Tenant Hosting.   Before starting, create two site collections that reside within the same web application.  

1. Create a subscription and assign sites to it:

$sub = new-spsitesubscription

$sub  

2. Pulling the site collection or set of site collections you wish to join to the site group:

​ get-spsite

$site = get-spsite | where {$_.url -eq "http://contoso"}

$site

​

​3. Add  the site collection $site, to the newly created site subscription $sub.

set-spsite -identity $site -sitesubscription $sub

Check whether it has been added correctly by doing the following:

get-spsitesubscription

If a database ID exists, then you can type the following

get-spdatabase | where-object {$_.id -match "full or partial guid"}

Will output the results of the associated site collection.

​

4. Create a secondary subscription and associate a different site collection within same web application for demonstration purposes using the steps above.

5. Create a SubscriptionSettings Service Application and Proxy

​A.  Start the WSS Subscription Settings Service

B. Create Service Application and Proxy via PowerShell

    $appPool  =  New-SPServiceApplicationPool -Name SettingsServiceApppool -Account domain\username

    $sa = new-spsubscriptionsettingsserviceapplication –Name SubscriptionSettingsServiceApplication –Databasename SubscriptionSettingsServiceApplicationDB –applicationpool appPool

    $sap = new-SPSubscriptionSettingsSericeApplicationProxy –ServiceApplication $sa

6. Creating the Tenant Admin Site for each site group

​$sub = get-spsitesubscription –identity “http://server”

$tasite = new-spsite –url “http://Contoso/sites/tasite1” –template “tenantadmin#0” –owneralias domain\username –sitesubscription $sub -AdministrationSiteType tenantadministration

7. Provision a search service application in hosting mode.  Please see the Provisioning Search Service application using Powershell blog.

Note:  You must append the –Paritioned parameter to the following cmdlets.

• new-spenterprisesearchserviceapplication
• new-spenterprisesearchserviceapplicationproxy

From Powershell, run “get-help cmd-let –full” for more details.

8. Feature's and hosting

Once a feature has been installed into the farm, it's available to all sites and can be activated through manage features pages.  Simply stated, all installed\deployed features to a specified web application will be available to all sites that reside under the associated web application.  This applies to both non-hosted sites “sites that don’t belong to a site group” and hosted sites “Sites that are associated and are members of a particular site group.  A farm administrator is highly unlikely going to want all custom features to be available to every site group.  For Example, a custom feature might need to be deployed to sites under Site Group 1 and another custom feature must be deployed to sites under Site Group 2.   In a hosting scenario, if a feature is catered to a particular site group, a Farm Administrator will not want to expose those features to other site groups.

The way to control this in a hosting scenario is you only provide the features available to a given site group through a series of  PowerShell commands.  Any features not listed are excluded and not available for all site collections that belong to the corresponding site group. 

In order control or restrict what features are available to an associated site group "site subscription", you can create what are called Feature packs. 

So the flow is the following:

1. Create a feature pack
2. Add features you want to be available for the associated site group
3. Assign the feature pack to the site group

Steps are the following:

A) Create a new site subscirption feature pack which creates a feature set limited to the specified site subscription.

$FP = new-spsitesubscriptionfeaturepack

B) Add the features you would like to add to the newly created subscription feature pack.

Get-spsitefeature

Copy the iD or id's of the features you want.

For example: 830b81f9-e008-41f4-b3c3-9946b7eb3578

Drop it in a variable now:

$feature = get-spsitefeature 830b81f9-e008-41f4-b3c3-9946b7eb3578

C) Add the feature has a member of the featurepack you created in step 1.

Add-SPSiteSubscriptionFeaturePackMember -Identity $fp -FeatureDefinition $feature

To confirm it took run $fp

D) Assign the featurepack to the site group "SiteSubscription"

Set-SPSiteSubscriptionConfig -identity $sub1 -featurepack $fp

Note:  Be careful what features you decide to leave out.   Some features are required in order to successfully create sites.   For Example, if a featurepack only contains one custom feature and that feature pack is assigned to a site group, The tenant administrator will be unable to create sites via the Tenant Administration web site associated to that paritcular site group.  Also, any owners of sites that reside under the site group will be unable to create subsites.  The following error will persists within the ULS logs:

04/05/2010 10:42:02.75    w3wp.exe (0x12FC)    0x02C0    SharePoint Foundation    Runtime    tkau    Unexpected    System.InvalidOperationException: Failed to activate feature 'BasicWebParts' (id: '00bfea71-1c5e-4a24-b310-ba51c3eb7a57').  The feature is not a member of the associated site subscription feature pack.    at Microsoft.SharePoint.SPFeatureCollection.AddInternal(SPFeatureDefinition featdef, Version version, SPFeaturePropertyCollection properties, Boolean force, Boolean fMarkOnly)     at Microsoft.SharePoint.SPFeatureCollection.AddInternalWithName(Guid featureId, String featureName, Version version, SPFeaturePropertyCollection properties, Boolean force, Boolean fMarkOnly, SPFeatureDefinitionScope featdefScope)     at Microsoft.SharePoint.SPFeatureManager.EnsureFeaturesActivatedCore(SPSite site, SPWeb web, String sFeatures, Boolean fMarkOnly)     at Microsoft.SharePoint.SPFeatureManager.<>c__DisplayClassa.<EnsureFeaturesActivatedAtSite>b__9()     at Microsoft.SharePoint.SPSecurity.RunAsUser(SPUserToken userToken, Boolean bResetContext, WaitCallback code, Object param)     at Microsoft.SharePoint.SPFeatureManager.EnsureFeaturesActivatedAtSite(Byte[]& userToken, Guid& tranLockerId, Int32 nZone, Guid databaseid, Guid siteid, String sFeatures)     at Microsoft.SharePoint.Library.SPRequestInternalClass.ApplyWebTemplate(String bstrUrl, String bstrWebTemplateContent, Int32 fWebTemplateContentFromSubweb, Int32 fDeleteGlobalListsWithWebTemplateContent, String& bstrWebTemplate, Int32& plWebTemplateId)     at Microsoft.SharePoint.Library.SPRequest.ApplyWebTemplate(String bstrUrl, String bstrWebTemplateContent, Int32 fWebTemplateContentFromSubweb, Int32 fDeleteGlobalListsWithWebTemplateContent, String& bstrWebTemplate, Int32& plWebTemplateId)     at Microsoft.SharePoint.SPWeb.ApplyWebTemplate(String strWebTemplate)     at Microsoft.SharePoint.ApplicationPages.TA_CreateSiteCollection.BtnCreateSite_Click(Object sender, EventArgs e)     at System.Web.UI.WebControls.Button.OnClick(EventArgs e)     at System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument)     at System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument)     at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)    c3b97351-e0bf-4351-a1bf-2e71c57fe2eb

Russ Maxwell, MSFT

Hosting 101 Part 1

​Before providing the steps of setting up hosting, it's important to understand the main concepts behind O14 hosting.

SharePoint 2010 has a rich set of hosting features which triumph over the previous version in many ways.  It's now simple to setup hosting on the site collection level. 

For Example:

• Tenant "customer 1" on site collection 1 is hosted on web application A
• Tenant "customer 2" on site collection 2 is hosted on web application A

Each tenant would only have full administrator rights on his/her assigned site collection.  Customer 2 and users accessing site collection 2 wouldn't be able to access Customer 1's site.

Also, service applications that are in hosting mode would keep each tenants data separate from another tenants.  For example, one shared search service application could service customer 1\site collection 1's data and customer 2/site collection 2's data while keeping them separate from each other.

For Example: Users searching in site collection 1 will not be able to search and find content that resides in site collection 2.  Users searching in site collection 2 will not be able to search and find content that resides in site collection 1.  They will be able to search and locate data within the site collection they are searching from.

Site Groups

The segmentation is possible through the use of site groups also known as site subscriptions.  In the example above, customer 1's site collection belongs to site group 1 and customer 2's site collection belongs to site group 2.

Things to know about site groups:

1. Sites can belong to only one site group at a time.

2. Sites cannot join a site group that contains sites that exists on a different web application.

3. A site group can span across more than one content database

4. No GUI interface for managing site groups.  PowerShell is required to create/manage/remove site groups.

Tenant Admin Site

A hosted customer is referred to as a tenant.  You can provision a tenant admin site which gives the tenant full administrator rights over the site collection.  The tenant admin site can be used to create additional sites for example after self service site creation is enabled.

Service Applications and Hosting

Be default, a service application is consumed at the web application level.  So all sites, under a web application would consume from the same service application and data would be shared.  In hosting mode, a shared service application partitions data where every site group has its own partition.  The partition isn't shared meaning other site groups wouldn't be able to see this data even though all sitegroups are using the same service application.  Configuring hosted service applications may differ based on the type of service application that is being deployed. 

For Example:  Deploying a Shared search service application requires you to use PowerShell with addition of -partitioned switch

Features

It's also possible to deploy features in an "a la carte" manner to site groups.  So one site group can have more features available than another.  This is configurable from PowerShell.

Hosting Part 2 provides a step by step walk though using PowerShell

-Russ Maxwell, MSFT

I recently had a customer that was using custom code against the OM in order to pull list items via the Lists.GetListItems() method. When attempting to run this method it was returning the following exception:

Microsoft.SharePoint.SoapServer.SoapServerException, No InnerExpection System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall) System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters) customApp.Lists.Lists.GetListItems(String listName, String viewName, XmlNode query, XmlNode viewFields, String rowLimit, XmlNode queryOptions, String webID) in E:\customApp\customApp\Web References\Lists\Reference.vb:test 403. CustomApp.Application.ImportObjects(ObjectBase& ob, String obj, String ouid, String otype, String site, String liid, String viid, String folder, String title, String usr, String pwd, String dom, String site2, String liid2, String modname, String semmodname, String lookupfield, Boolean bImportClass) in E:\\customapp\App.vb:1111.

This particular list had 10 lookup columns pointing to 10 different lists. This specific behavior is by design because we put a cap via “list throttle” settings at the web application level.

This can be accessed via Central Administrator\Application Management\Manage Web Applications\General Settings\Resource Throttling:

As noted above, the default value for default lookup fields is 8. Once this was increased to 10 or higher, the custom code returned the items successfully and no more soap exception. I just wanted to point this out as I think many developers will utilize Lists.GetListItems() and need to be aware of resource throttling..

Note: Other settings reside in Resource Throttling like “List View Threshold”, which defaults to 5,000 items.

See this blog for good information on that setting:

http://blogs.technet.com/speschka/archive/2009/10/27/working-with-large-lists-in-sharepoint-2010-list-throttling.aspx