SharePoint Brew

Why are my blog post showing up in different places?

2009-12-18T00:38:00Z

I will be moving a good portion of my blogs as well as any new ones over to our new SharePoint Escalation Services Team blog. We think it’s a good idea to get our blogs more consolidated. This also ensures that blog content is mostly coming from Microsoft Support Escalation Engineers and Escalation Engineer’s.

Check out the blog here:

http://blogs.msdn.com/spses

I’ll still be posting my blog content here as well but just wanted to put this on your radar…

SharePoint 2010 Logging Improvements – Part 1

2009-12-18T00:18:00Z

Part 1 of this blog will describe ULS and Event log improvements and what you need to know. The new changes simplify troubleshooting and provide the Administrator more flexibility in managing different aspects of ULS logging. I will expand on this topic in Part 2 and will discuss Developer Dashboard. A special shout-out goes to Dan Winter for the knowledge transfer and technical review.

Correlation ID

A deep investment has been made to improve Correlation ID in SharePoint 2010. Events such as page loads and other common events within ULS logs have a corresponding Correlation ID. This ID is grouped with other events based on the same conversation. For example, User A hits a WFE and accesses a site. All the events in the ULS logs that correlate with User A’s login all share the same correlation ID. This makes mining the ULS logs easy in that you can filter based on correlation ID. If the conversation hits multiple servers in the farm, those other servers maintain the exact same correlation ID. A common scenario for this is when a WFE is calling a web service on an application server. It’s also possible to filter on Correlation ID via a SQL profiler trace so you can truly get a farm wide picture of what happened at each level of the conversation. SharePoint error pages also expose a correlation ID along with time stamp of the error so now it’s simple to find the conversation in the servers corresponding ULS logs.

For Example:

Event Throttling

ULS and Event logging now has a richer category based management in that you can set specific subcategories to a unique level of logging while throttling other subcategories that belong to same root category.

For Example, troubleshooting query processor, turn up verbose on query processor component:

In the example above, the root category is bold it flags an administrator when unique logging has been set within a root category. Also, note the different logging levels which can be set on each sub category.

When the issue has been resolved or custom logging levels are no longer required, you can now set logging back to default:

Log File Retention

In SharePoint 2007, the default log file retention policy was to set log files to generate once every 30 minutes with a total of 96 log files at any given time. This equates to 48 hours. ULS logging has been improved by reducing log file size by at least 50 %. This allows a completely new approach to setting log file retention policy. The default setting is now at # of days with the default being 14. A new option exists to restrict trace logs to a fixed disk size. This is unchecked by default.

For Example:

Once day 15 hits, log files from day 1 are pruned. If restrict trace log setting is enforced to a fixed disk size, then both settings are enforced. The order on which the settings are enforced is dependent upon which condition is hit first.

For Example:

Number of days to store log files: 14

Restrict trace log disk space usage: 2 GB

SharePoint administrator is chasing a problem and cranks up some verbose logging due to profile provision problem. This causes the total ULS logs size to hit 2 GB on day 3. In this case, the oldest logs are pruned to honor the restrict trace log setting. These settings are configurable via the Set-SPDiagnosticConfig cmdlet.

Note: The highest level you can set the trace log disk space usage is 1 TB.

Event Log Flood Protection

The name says it all. This feature will prevent one event from flooding the event log. This is enabled by default within diagnostics logging:

With this option enabled, if the same event fires 5 times within 2 minutes, we will suppress the event from logging and throw a summary event at the 2 minute mark with a nice description of how many times the event actually fired. This is the default behavior and can be customized using PowerShell via the Set-SPDiagnosticConfig cmdlet.

Note: To qualify has the same event is an event with the same event source and ID. The event severity and/or text doesn’t apply and is not checked.

Manage logging using PowerShell

A more hardcore SharePoint administrator is likely to use PowerShell for all administration including managing ULS logs. The PG has given us a very rich set of command-lets to do things like get\set diagnostic levels and retention policies. It’s possible to mine ULS logs using PowerShell for monitoring or troubleshooting purposes. I’ll discuss each area below and provide some examples.

Retrieve logging levels

The following cmdlet retrieves logging levels:

Get-SPLogLevel – Displays a list of objects or diagnostic levels

The cmdlet pulls all logging levels but what if you wanted to filter this result by area or specific category?

Examples:

To retrieve all logging categories by area “WSS Search Diagnostics”:

Get-SPLogLevel –identity “WSS Search Diagnostics:*” | FL

To retrieve a specific logging category like Query Processor within the WSS Search Diagnostic area:

Get-SpLogLevel –identity “WSS Search Diagnostics:Query Processor” | FL

Set logging levels

The following cmdlet sets logging levels:

Set-SPLogLevel - Sets trace logging and event logging levels to a category

Note: No parameters are set to required but you still need parameters to correctly set logging levels

Examples:

To set the entire area “WSS Search Diagnostics” to event log level of monitorable and trace level to verbose:

Set-SPLogLevel -TraceSeverity Verbose -EventSeverity Warning –Identity “WSS Search Diagnostics:*”

It’s possible set a specific category within an Area. For Example, setting Query Processor to verbose within WSS Search Diagnostics:

Set-SPLogLevel -TraceSeverity Verbose -EventSeverity Warning –Identity “WSS Search Diagnostics:Query Processor”

Note: Logging levels can be reset back to default level using Clear-SPLoglevel

Creating and Merging Log files

It’s possible to force a new uls log to be created by running new-splogfile. This will end the current log file and start a new one. This is valuable when attempting to reproduce a problem and want to minimize the total size of the log file. Merge-SPLogFile combines trace log entries from all farm computers into a single log file on the local computer. It does this by creating a one-time timer job and executing it to create the merged log. This is extremely valuable if you are tracking an issue and suspect it’s hitting multiple servers in a farm. Since the conversation across multiple servers will contain the exact same correlation ID, the merge-splogfile becomes a powerful cmdlet for troubleshooting.

Some examples of Merge-SPLogFile:

Merge-SPLogFile -Path "C:\Logs\FarmMergedLog.log" -Overwrite

The preceding example merges the last hour of log data from all farm computers with no filtering.

Merge-SPLogFile -Path "C:\Logs\FarmMergedLog.log" -Overwrite -Area Search

The preceding example merges the last hour of log data from the Search area

Additional parameters are available to tune results. For Example, looking for something specific in the ULS message you can filter results using the Message parameter.

Mining ULS Logs via PowerShell

It’s possible to view ULS log entries using the Get-SPLogEvent cmdlet. At beta 2, Running this cmdlet by itself retrieves every event from every ULS log within the directory hosting ULS logs. Some examples of running Get-SPLogEvent:

Example 1: ULS log or logs are moved to a different directory named “newpath”, run the following:

Get-SPLogEvent –Directory “C:\newpath”

Example 2: Retrieve all the events within a certain time period run the following:

Get-SPLogEvent -StartTime "12/04/2007 17:00" -EndTime "12/04/2007 18:00"

Example 3: Retrieve the first 10 events and last 2 events from logs directory:

Get-SPLogEvent | select –first 10 –last 2

Example 4: Retrieve all events with same correlation ID:

Notice the message: “Warning: column “Message” does not fit into the display and was removed.

You can include the message by performing the following:

Note: The more columns you include, the more difficult it is to read the message because it wraps.

Example 5: Retrieve all events based on a specified level.

Get-SPLogEvent | ?{$_.level –eq “critical”}

Tips for User Profile Service Application provisioning with SharePoint 2010 Beta 2

2009-12-09T21:04:53Z

This is just another blog of many discussing some tips to provisioning User Profile Service Application. I refer to User Profile Service Application as UPA in this blog. The bottom of this blog will contain some links to walk you through the process. Things have improved greatly in Beta 2 but you can get caught up in frustrating problems provisioning the UPA. This can often lead to serious confusion on what exact account permissions are required for successful UPA provisioning. This blog is specifically written after testing with Beta 2. At RTM, this may not be 100 % accurate.

The basic steps are the following:

1. Provision a UPA either through the Farm Configuration Wizard or from Manage Service Application page within Central Administrator.

2. Start the "User Profile Synchronization Service" within Central Administrator\System Settings\Manage services on server

Note: This must show in a started state before continuing

3. Within Central Administrator, Application Management, and select User Profile Service Application. Click Manage button from the Ribbon.

4. Click Configure Synchronization Connections and click Create New Connection

5. Fill out the fields and for Forest Name specify the FQDN of the DC you want to use. Keep it on Auto discover domain controller. The account name specified must have "Replicate Directory Changes" permission. Populate Container and choose which OU's that will participate and hit OK.

Permission Requirements for UPA provisioning

The next hot topic is what account should I use to provision a UPA and what permissions are required? At beta 2, you must use the same account used when you installed SharePoint 2010. This is referred to as the farm service account. This account is automatically added to the Central Administrator’s application pool identity and will run the timer service. If you’re not sure which account is the farm service account, then the easy way to verify is to look at the Central Application Pool identity or account used to run the timer service.

Here is a quick re-hash of the account requirements before starting UPA provisioning:

1. User account is the same account that was used to install SharePoint 2010

2. User account is a member of Farm Administrator group within Central Administrator

3. User account is added to local administrator group

4. User account is added to sysadmin role on SQL server

5. User account is allowed by policy to log on locally to the server where Profile Synchronization will be deployed

6. User account is at least member of the domain’s Pre-Windows 2000 Compatible Access group.

Note: If the user is a domain administrator, membership to this group isn’t required

7. User account specified to perform a profile import requires "Replicate Directory Changes" permission.

For more information on this see the following:

http://blogs.msdn.com/sharepoint/archive/2009/11/18/path-to-user-profile-synchronization-success-in-sharepoint-2010-beta.aspx

PowerShell Intro for SharePoint Administrators - Part 2

2009-11-19T21:05:00Z

PowerShell has a rich set of advanced features that provide an Administrator with a multitude of options depending on what they would like to do. This nugget will take a more focused look into the many ways objects can be used. Remember, an object in this case is a variable. From Part 1 of this nugget, you know that variables are objects that hold something. This nugget will conclude with some PowerShell scripting basics. Thanks goes to Sheyi for providing a technical review.

PipeBind

PipeBind is a concept where a parameter accepts an object “variable” of a specific type. If you look at the syntax of get-help new-spsite –full, you will discover some parameters accept pipe bind:

New-SPSite -Url <String> -OwnerAlias <String> [-AssignmentCollection <SPAssignmentCollection>] [-Confirm <SwitchParameter>]] [-ContentDatabase <SPContentDatabasePipeBind>] [-Description <String>] [-HostHeaderWebApplication <SPWebApplicationPipeBind>] [-Language <UInt32>] [-Name <String>] [-OwnerEmail <String>] [-QuotaTemplate <SPQuotaTemplatePipeBind>] [-SecondaryEmail <String>] [-SecondaryOwnerAlias <String>] [-SiteSubscription <SPSiteSubscriptionPipeBind>] [-Template <SPWebTemplatePipeBind>] [-WhatIf [<SwitchParameter>]] [<CommonParameters>]

In this example, there are several parameters that accept pipe binds. Here is an example of passing a variable to the template parameter which accepts a pipe bind.

$tem = get-spwebtemplate –identity STS#0

New-spsite –url http://contosoweb –owneralias contoso\administrator –name contososite –template $tem

Properties and Methods

Variables have properties and method’s available based on the type of the variable. For example, declaring a variable and assigning it a web application will contain all properties and functions for type SPWebApplication. To view all of the available properties and methods, you can pipeline the get-member cmdlet.

For Example:

$web = get-spwebapplication

$web | get-member

Note: This is not a complete list of all method’s and properties available for this type.

It’s possible to add conditions to get more desirable results. For Example, you can use the where alias if you just want to see a specific member type. To view only members of type Method punch in the following:

$web | get-member | where {$_.MemberType –eq “Method”}

Note: It is possible to do more than one pipeline J

Viewing Properties

Before updating and viewing specific properties of an object, it’s important to know how to first view all properties and there corresponding values. By default, only a few properties are exposed by simply printing a variable.

For Example:

Use get-object cmdlet to view all properties of a given object. An alias for the get-object cmdlet is available called select which is what I prefer to use.

For Example:

Note: Another option to view all properties of a given object is available.

For Example: $web | fl

If you happen to know which property to specify, simply append the property name with the variable.

For Example:

In this example, two content databases are outputted because the Web Application has two content databases attached. What if you only need to make changes to one of them? Where-object command-let is available which assists in filtering down results. An alias exists for the where-object command-let called where which is what I prefer to use.

$web.contentdatabases | where {$_.name –eq “WSS_Content”}

The example above demonstrates how to view and filter properties against a single object\variable. What if you know which property to use and simply want to filter directly from the command-let. Some command-lets offer parameters which allow filtering on the fly.

For Example, the –filter parameter is available running Get-SpSite command-let.

Get-spsite –filter {$_.owner –eq “contosoweb\administrator”}

This example outputs every site collection where the owner is set as contosoweb\administrator. This is a much faster query when dealing with a large # of sites.

Note: You can also use the –like wild card instead of –eq.

For Example: Get-spsite –filter {$_.owner –like “contoso\ad*”}

Setting Properties

Besides viewing properties, it’s possible to update properties as well.

For Example, setting a property called name and updating it using update method:

$web.name

Output: SharePoint – 80

$web.name = “My Renamed Site!”

$web.update()

$web.name

Output: My Renamed Site!

Running Methods

Besides using the update method above to set a property, you have a variety of methods available for a specific object type. For Example:

Scripting with PowerShell

PowerShell scripts provide an automated approach to group one or more command-lets in order to accomplish one or more tasks. For Example, PowerShell scripts can provision new Web Applications\Site Collections, service applications, or backup\restore farm. Again, the possibilities are endless depending on what an Administrator wants to do. A powershell script is as easy as dropping your command-lets in notepad and saving the file with a .ps1 extension. Variables are often used in PowerShell scripts because they save time in writing scripts as well as provide a cleaner look to the script. The following example is a simple script to provision a new site collection:

$tem = get-spwebtemplate –identity STS#0

$russmax = get-spuser –identity contoso\administrator -web http://contosoweb

$DB = get-contentdatabase

new-spsite -url http://contosoweb/sites/dancan -template $tem -Name DanCanSite -Description DQ -owneralias $russmax -contentdatabase $db

After saving this script as newsite.ps1, launch powershell and run the following from the directory where the script resides:

.\newsite.ps1

Also, custom command-lets and functions can be created and used within PowerShell scripts to achieve a more desirable result. Advanced PowerShell scripts for SharePoint 2010 will be available on codeplex soon.

http://www.codeplex.com/SharePointPSScripts/

PowerShell Intro for SharePoint Administrators - Part 1

2009-11-11T22:27:00Z

PowerShell has been nicely integrated into SharePoint 2010 and is the eventual replacement of stsadm. Stsadm still exists with SharePoint 2010 for backward compatibility reasons but PowerShell is here to stay and will be the most widely adopted and used out of the two. PowerShell is included as a prerequisite which is automatically downloaded and installed during the prerequisite install of SharePoint 2010. Part 1 of this nugget will focus on basics of PowerShell including features and where to start. Part 2 will contain some more advance topics such as scripting and manipulating objects by setting properties or running methods against them.

Basics

Windows PowerShell™ is a task-based command-line shell and scripting language designed especially for system administration. Built on the .NET Framework, Windows PowerShell™ helps IT professionals and power users control and automate the administration of the Windows operating system and applications that run on Windows.

Built-in Windows PowerShell commands, called cmdlets, let you manage the computers in your enterprise from the command line. Windows PowerShell™ providers let you access data stores, such as the registry and certificate store, as easily as you access the file system. In addition, Windows PowerShell™ has a rich expression parser and a fully developed scripting language.

Windows PowerShell™ includes the following features:

Cmdlets for performing common system administration tasks, such as managing the registry, services, processes, and event logs, and using Windows Management Instrumentation.
A task-based scripting language and support for existing scripts and command-line tools.
Consistent design. Because cmdlets and system data stores use common syntax and naming conventions, data can be shared easily and the output from one cmdlet can be used as the input to another cmdlet without reformatting or manipulation.
Simplified, command-based navigation of the operating system, which lets users navigate the registry and other data stores by using the same techniques that they use to navigate the file system.
Powerful object manipulation capabilities. Objects can be directly manipulated or sent to other tools or databases.
Extensible interface. Independent software vendors and enterprise developers can build custom tools and utilities to administer their software.

Note: The basics section was taken directly from technet because I couldn’t have written this any better. See the resources section at the bottom of this blog for direct access to the site.

PowerShell Features

TAB – Expansion

It’s possible if you do not know an entire cmdlet to tab through what you have typed in order to find a match. Cmdlets always start with a verb-cmdlet name combination. Tab expansion is great for quickly tabbing through all matching cmdlets. For Example, the entire first part of the name (the verb) and the hyphen that follows it must be inputted. For example, if you type get-co and then press the Tab key, Windows PowerShell will automatically expand this to the Get-Command cmdlet. If you press Tab key again, Windows PowerShell replaces this with the only other matching cmdlet name, Get-Content.

Pipelining

Pipelining is the process where the output of one command is piped to a second command using the Pipline operator |.

For Example:

get-spwebapplication –identity http://contosoweb/ | new-spcontentdatabase –name contosodb

In this example, the output of get-webapplication is piped over to new-spcontentdatabase cmdlet. The new-spcontentdatabase cmdlet has –webapplication parameter and it’s required. You don’t need to specify this parameter because it’s piped over.

Format List

Format-List cmdlet formats the output of a command as a list of properties in which each property is displayed on a separate line. Typically, the alias of FL is used instead via piping.

For example: get-spwebapplication –identity http://contosoweb | FL

Aliases

An alias is another name you assign to a cmdlet, function, script, etc…. There are some built-in aliases within PowerShell. You can look at those by running the get-alias cmdlet. It’s possible to create aliases by using the Set-Alias cmdlet.

For Example, create an alias named DanCan which runs get-spwebapplication:

Set-Alias DanCan get-spwebapplication

Now running DanCan will produce the same result as get-spwebapplication.

Variables

PowerShell contains basic programming principles such as using Variables. Variables are simply an object that holds something. For Example, you can create variables to hold a specific content database. What you do with that variable is endless in terms of options available which will be discussed in the next nugget. For now, it’s important to understand how to declare variables.

To declare a variable named Var to hold content database named contosoDB type the following:

$var = get-spcontentdatabase –identity contosoDB

Type $var and it will output what’s stored in the variable.

Getting Help with get-help

PowerShell is accessible via the start menu:

Note: PowerShell is referred to as SharePoint 2010 Management Shell in SharePoint 2010

Scenario:

Great, it’s opened now what do I do! In this scenario, an administrator, Dan, must use PowerShell to create a new site collection using team site template and is unsure of which cmdlet or syntax to run.

The Get-Help cmdlet will output every alias, cmdlet, and function using a prefix of *. It’s great for finding out which cmdlet to run and get help on how exactly to run the cmdlet.

For Example: get-help *

Try it out and you’ll see it dump out lots of stuff like aliases and cmdlets. If you want to dump out just cmdlets you could type the following:

Get-help * | where {$_.category –eq “cmdlet”}

All SharePoint cmdlets start with SP so to output only SharePoint cmdlets you type the following:

Get-help *-SP* | where {$_.category –eq “cmdlet”}

This is great but I’d rather pipe it out to txt file. To pipe any output to a text file append > c:\filepath\filename.txt

Get-help *-SP* | where {$_.category –eq “cmdlet”} > c:\odst\output.txt

Note: Don’t worry about understanding the entire syntax, it will be more familiar after reviewing both PowerShell nuggets.

Verbs are appended to cmdlets which describe the action taken. For Example: creating, removing, or setting a new/modified value on an object. Since I’m looking to create I could be even more granular by typing the following:

Get-help new-SP* | where {$_.category –eq “cmdlet”}

After some mining, the administrator Dan needs to use the New-SPSite cmdlet. In order to properly run this command Dan needs to know the required parameters and the correct syntax. By default, running the following provides a brief summary and outputs syntax:

Get-help new-spsite

What it doesn’t tell you is which parameters are required and a brief description of each parameter. The following gives you that information:

Get-help new-spsite -full

A partial of the output:

SYNOPSIS
Creates a new site collection at the specified URL.

SYNTAX
New-SPSite -Url <String> -OwnerAlias <String> [-AssignmentCollection <SPAssignmentCollection>] [-Confirm <SwitchParameter>]] [-ContentDatabase <SPCon tentDatabasePipeBind>] [-Description <String>] [-HostHeaderWebApplication <SPWebApplicationPipeBind>] [-Language <UInt32>] [-Name <String>] [-OwnerEmail <String>] [-QuotaTemplate <SPQuotaTemplatePipeBind>] [-SecondaryEmail <String>] [-SecondaryOwnerAlias <String>] [-SiteSubscription <SPSiteSubscriptionPipeBind>] [-Template <SPWebTemplatePipeBind>] [-WhatIf [<SwitchParameter>]] [<CommonParameters>]
DETAILED DESCRIPTION

The New-SPSite cmdlet creates a new site collection with the URL and owner specified by the Url and OwnerAlias parameters.

PARAMETERS
-Url <String>
Specifies the URL that the new site collection uses. If this is not a host header site, the URL must start with the containing the Web application URL
        Required?                    true
        Position?                    1
        Default value
        Accept pipeline input?       True
        Accept wildcard characters? false

The required parameters are URL and Owner. Dan also wants to specify a team site template so it will look like the following:

New-spsite –url http://contosoweb/sites/snackattack -OwnerAlias contoso\farmadmin -template STS#0

Note – To get a full list of installed templates run the following: get-spwebtemplate

TIP: Get-Command cmdlet –syntax can also if an Administrator just needs to review the syntax.

Resources:

http://technet.microsoft.com/en-us/library/bb978526.aspx

http://technet.microsoft.com/en-us/library/dd347700.aspx

http://technet.microsoft.com/en-us/library/dd347728.aspx

http://technet.microsoft.com/en-us/library/dd347730.aspx

SharePoint 2010 Granular Backup-Restore Part 2

2009-10-22T16:40:00Z

This is part 2 of granular backup/restore blog in which I’ll cover the Recover data from an unattached content database option in Central Administration. This is located in backup/restore section under granular backup/restore. SharePoint 2010 utilizes SQL snapshots both to create and to restore from. This blog will also cover the overall uses of snapshots since they can be used in multiple ways. You can be has granular as possible in choosing what you want to restore from snapshots whether it’s the entire site collection or a specific document library\list.

Snapshot Basics

Before going though the steps it’s important to know the basics of SQL snapshots. SQL snapshots were introduced in SQL 2005 timeframe and only available to Enterprise and Developer editions of SQL. SQL 2008 maintains this rule in that snapshots are only available to Enterprise and Developer editions of SQL. SQL snapshots are a read-only copy of a data base as it existed at snapshot creation. Snapshots operate at the DB page level so when a page is about to be modified for the first time in the source DB, the page is first copied to the snapshot thus preserving the data record. If the page already exists and has been modified post snapshot, it’s not updated or copied to the snapshot. At snapshot creation, all pages from source db are considered new and are copied to the snapshot. Snapshots are linked to the source database where they originated. If the source database goes offline for any reason, then the snapshot is unavailable. This is why snapshots are great for granular backup/restore operations but this shouldn’t be your main backup/restore method. Keep in mind that snapshots are limited to content databases only. To learn more about snapshots check out the following:

http://msdn.microsoft.com/en-us/library/ms175158.aspx

What can SharePoint 2010 Administrators do with snapshots?

SharePoint 2010 Administrator can now do common administrative tasks when it comes to snapshots. The following lists some common things which can be performed on a SharePoint 2010 server using PowerShell.

· Create snapshots

· Delete snapshots

· Restore snapshot

· Use snapshots for granular recovery operations <- Can also use Central Administration

Note: You cannot create, delete, or restore snapshots on SharePoint 2010 standalone server due to the fact it runs on SQL 2008 express edition. Snapshot operations are not supported against SQL server express edition.

Creating Snapshots

Two common methods exist for creating snapshots of SharePoint 2010 data.

Method 1: Using SQL Server

Snapshots are created via TSQL commands within SQL Management Studio query window. SQL Management Studio doesn’t expose any UI for creating snapshots. If my content database is named "WSS_Content" then I would run the following query from SQL Management Studio:

CREATE DATABASE ContentSnapshot on (NAME = "WSS_Content", Filename = 'c:\snapshot\contentsnap.ss') as SNAPSHOT OF "WSS_Content";

Once this is completed, the snapshot resides in the snapshots folder.

Method 2: Using PowerShell aka “SharePoint Management Console”

SharePoint Management Console can also be used to create snapshots. Yes, this is a fancy way of saying powershell but get used to hearing it since it’s listed that way on the start menu. You can now create SQL snapshots from a SharePoint 2010 server. Using SharePoint Management Console you can input something like the following:

$ContentDB = get-spcontentdatabase WSS_Content

$ContentDB.Snapshots.CreateSnapshot()

Below, a new snapshot is created within SQL Management Studio:

Recover Data from an unattached content database

The recover data from an unattached content database option is used for pulling data out of a snapshot or detached content database and exporting to a file. The exported file can be imported back into production using the sp-import cmdlet via powershell. Both Central Administrator and Powershell can be used to pull data out of a snapshot. Three specific operations exist depending on what specifically you want to pull out of the snapshot. Selecting the recover data from an unattached content data base option exposes the following operations.

Browse Content – Provides ability to browse for a specific site collection, Site, or list. Once a selection is made, the same operations are available which is backup site collection or Export site or list.

Backup Site collection – Provides the ability to backup a site collection from a snapshot which can then be restored into production

Export Site or List – Provides the ability to export a specific site, list, or document library from a snapshot

Walkthrough

Using Granular backup/restore to pull a document library from a snapshot and import into production

Using Central Administrator

This example demonstrates exporting a document library named hr out of a snapshot named “contentsnapshot2”:

1.) Within Central Administration, Select backup/restore, Recover Data from an unattached content database

2.) The following screen is where the SQL server name is populated and the corresponding snapshot name:

After filling out required fields and selecting Export site or list click next.

3.) After clicking next, drill down to specified list as well as includes options for exporting security, version, and specify export path.

Select the start export button to pull the data from the snapshot into the export.cmp.

4.) Finally, run the import-spweb cmdlet via powershell to import the data back into production. For these steps, see the previous nugget.

Using Powershell

This example demonstrates exporting a document library named hr out of a snapshot named “contentsnapshot”:

1.) Get the snapshot database and assign it to variable named $snappy:

$snappy = get-spcontentdatabase -ConnectAsUnattachedDatabase -DatabaseServer contososql -DatabaseName contentsnapshot

2.) Pipe $snappy with export-spweb cmdlet to pull hr list out of the snapshot:

$snappy | export-spweb http://contosoweb –usesqlsnapshot –itemurl /hr –path \\contososql\bu\mybackup.cmp

3.) Finally, run the import-spweb cmdlet via PowerShell to import the data back into production. For these steps, see the part 1 of granular backup/restore blog.

SharePoint 2010 Granular Backup-Restore Part 1

2009-10-21T20:44:00Z

Hello! Russ Maxwell here and I’d like to provide a glimpse into SharePoint 2010 granular backup/restore. Several things have changed and have been improved in this area. This article is specifically themed around granular backup\restore and what you need to know.

Key Concepts:

Granular Backup: Granular backup has been placed into its own section within Central Administrator. This includes multiple options including the following options:

· Perform a site collection backup

· Export a site or list

· Recover data from an unattached content database

Basics

In SharePoint 2007, all granular backup and restore operations were only available using stsadm. SharePoint 2010 has integrated granular backup restore operations into both Central Administrator and PowerShell. Some backward compatibility does exist with stsadm but PowerShell is the new and improved replacement. Stsadm will be left out of this blog. Part 1 of this blog will include all granular backup/restore options except for “Recover data from an unattached content database”. Part 2 of this series will be dedicated to that topic. Performing backup operations deemed as granular can be performed from Central Administrator or PowerShell. Granular restore operations are only available using PowerShell. SharePoint 2010 is more flexible in terms of what can be backed up and restored. It’s possible to backup and restore site collection, sites, lists, document libraries, and items. The options for performing granular backups using Central Administrator are:

· Perform a site collection backup

· Export a site or list

· Recover data from an unattached content database (Covered in Part 2 of this series)

Granular backup operations

Backing up a Site Collection

You can backup a specific site collection using either Central Administrator or Powershell.

Using Central Administrator

Select Backup and Restore
From the UI select Perform a site collection backup
Select the appropriate site collection from the pull down menu
Include proper UNC path including backup file named like: filename.bak
Selecting Start backup redirects to _admin/sitebackuporexportstatus.aspx page.

Using Powershell

backup-spsite -identity http://contosoweb -path \\server\backupshare\bufile.bak

Backing up a Site, Library, or List

Exporting a Site/sub-site

Export a specific site using Central Administrator:

1. Select Backup and Restore
2. From the UI select Export a site or list
3. Select Site Collection\Site "leave list drop down clear"
4. Optional categories is to export full security or export versioning history
5. Selecting Start export redirects to _admin/sitebackuporexportstatus.aspx page.

Using Powershell

export-spweb -identity http://contosoweb -path \\servershare\bu\site.cmp

Export a specific list or library:

1. Select Backup and Restore
2. From the UI select Export a site or list
3. Select Site Collection\Site
4. Select a list
5. Optional categories “export full security or export versioning history”
6. Selecting Start export redirects to _admin/sitebackuporexportstatus.aspx page.

Using Powershell

export-spweb -identity http://contosoweb -path \\servershare\bu\hrpowershell.cmp -itemurl /cake

Note: This example demonstrates exporting a document library named cake from Contosoweb site.

Granular restore operations

Granular restore operations require the use of Powershell as no UI in Central Administrator exists. Two cmdlets are available for granular restore operations.

restore-spsite

This cmdlet is used to restore site collections that have been backed up using “Perform a Site Collection Backup” in central administrator or backed up using backup-spsite cmdlet.

For Example: restore-spsite -identity http://contosoweb -path \\server\backupshare\backupfile.bak

You have plenty of parameters available but the example demonstrates the required parameters. To get the full scope of parameters you can run:

get-command restore-spsite -syntax

What you should know about using restore-spsite

The identity/URL parameter shouldn't point to a site collection that is present. If the intention is to overwrite an existing site collection, you must use the -force parameter.

import-spweb

This cmdlet is used to restore sites, list, and libraries exported with “export site or list” option in central administrator or using export-spweb cmdlet.

For Example: import-spweb -identity http://contosoweb/ -path \\appserver\bu\list.cmp

Note: You can’t decipher if a list or library is being imported into the contosoweb site using import-spweb. This has been predefined in the backup prior to running import-spweb.

What you should know about using import-spweb

Sites and subsites: Importing a site/subsite works as long as you specify a URL with a site that contains a matching template. For Example, importing a team site to http://contosoweb requires that I first provision a site using the team site template using URL http://contosoweb. Attempting to import to a site using a blank template or different template will result in an error.

List and Libraries: It’s important to determine exactly what items you want to import into a list or document library. If no items exist, then running the command in the example above will suffice and all items will be imported. What if a user accidently deletes 5 items out of 1,000 and you simply want to restore those 5 items? By default, the import-spweb cmdlet overwrites items if they exist with the restored version. In this case, the item version prior to restore can be salvaged by restoring the previous version as long as versioning is enabled. To prevent a scenario where you want items which currently exists to be ignored by import operation, you must include the -updateversions parameter with ignore

So in this example, to restore 5 items without affecting the 995 items which currently reside in a list named odst I can run the following:

Import-spweb –identity http://contosoweb/ -path \\appserver\bu\odst.cmp -updateversion ignore

Note: New Items created within the ODST library after initial export will not be touched by an import operation.

Check Granular backup job status

A sleek new page has been added to check on granular job back status. Two ways to get to this page is through Central Administrator and after initiating a granular backup automatically redirects you to the _admin/sitebackuporexportstatus.aspx page.

Stay tuned for Part 2 of this series which I’ll discuss how SharePoint 2010 leverages snapshots using the Recover data from an unattached database option.

SharePoint 2010 Installing prerequisites without an internet connection.

2009-10-21T18:45:00Z

How to automate prerequisite install without an internet connection

I want to cover a few things about the prerequisite installer that everyone should know about. First, all but one prerequisite component is installed locally. The rest of them are downloaded from the web during setup. This is great for servers with an internet connection but what if you want to install prereq’s without an internet connection. This blog demonstrates how to run automate the install of prerequisite components on a box without requiring an internet connection.

First, prerequisites need to be manually downloaded and moved over to server:

SQLClient http://go.microsoft.com/fwlink/?LinkId=123718

SyncFramework http://go.microsoft.com/fwlink/?LinkID=141237

MSChart http://go.microsoft.com/fwlink/?LinkID=122517

PowerShell http://download.microsoft.com/download/D/0/E/D0E6D2C1-2593-4017-B26D-7375BC9263D5/PowerShell_Setup_amd64.msi

Geneva Framework http://download.microsoft.com/download/F/3/D/F3D66A7E-C974-4A60-B7A5-382A61EB7BC6/MicrosoftGenevaFramework.amd64.msi

Microsoft ADOMD.NET http://download.microsoft.com/download/A/D/0/AD021EF1-9CBC-4D11-AB51-6A65019D4706/SQLSERVER2008_ASADOMD10.msi

Automate Installing Prerequisites without Internet Connection

It's possible to install each prerequisite manually on a server with no internet connection but a more automated option exists. The steps are the following:

1.) Place the downloaded files into a directory. For my example: C:\sp

2.) Run the following command via command prompt within the root directory of the install media:

PreRequisiteInstaller.exe /SQLnCli:C:\sp\sqlncli.msi /ChartControl:C:\sp\MSChart.exe
/PowerShell:C:\sp\PowerShell_Setup_amd64.msi /Sync:C:\sp\Synchronization.msi
/IDFX:C:\sp\MicrosoftGenevaFramework.amd64.msi /adomd:C:\sp\SQLSERVER2008_ASADOMD10.msi

SharePoint 2010 Configuring Search Service Application using PowerShell

2009-10-20T18:29:00Z

NOTE: This has been updated to work with Beta 2 build!

It might be necessary at some point to use PowerShell to provision search service applications. For Example, setting up a search service application for hosted sites requires you to use PowerShell. The following steps manually take you through this process and I highly recommend going through the steps to become more familiar with the command-lets.

Creating Search Service Application using PowerShell

1. Create Application Pool

Creating a an application pool for your search service application and throwing the object into a variable called $ app:

$app = new-spiiswebserviceapplicationpool –name contososearch-apppool –account domain\user

2. Create search service application

$searchapp = new-spenterprisesearchserviceapplication -name ContosoSearchServiceApplication -applicationpool $app

Note: Add the -partitioned switch after -name if the search service application will be consumed in a hosted environment.

3. Create search service application proxy

$proxy = new-spenterprisesearchserviceapplicationproxy -name Contososearchserviceapplicationproxy -searchapplication $searchapp

Note: Add the -partitioned switch if the search service application will be consumed in a hosted environment.

Verify the search service application proxy is online. It should be online by default..

$proxy.status

If it's not online, you can change the status by punching in the following:

To change this property you could type something like this:

$proxy.status = “online”

Finally, you must update the change by calling the update method.

$changestatus.update()

4. Provision Search Administration Component

Configure the administration component of the associated Searchserviceapplication. You can do this with the following steps:

$si = get-spenterprisesearchserviceinstance –local

set-spenterprisesearchadministrationcomponent –searchapplication $searchapp – searchserviceinstance $si

5. Provision Crawl and Query Components

By default, a search application created in PowerShell has a crawl topology but is missing the following:

· crawl component

· query component

You cannot add a crawl\query component to the default crawl\query topology because it's set as active and the property is read only. The easiest way around this is creating a new crawl topology and new query topology. After creating both, they will be set as inactive by default. This allows for both crawl components to be added to crawl topology and query component to be added to newly created query topology. Finally, you can set this new crawl topology to active.

Steps are the following:

a. Create Crawl Topology

Create object and input output of newly created crawl topology.

$ct = $searchapp | new-spenterprisesearchcrawltopology

What if you already created a new crawl topology but never dropped it in a variable. That's okay, you can still drop it in a variable. First, you must know the ID for the crawl topology which should still be marked as inactive.

$searchapp | get-spenterprisesearchcrawltopology

Let's say the ID in this case is 2e25dcbf-ee71-41e4-b843-3f17d1064386.

$ct = $crawltopology = $searchapp | where-object {$_.id -eq "2e25dcbf-ee71-41e4-b843-3f17d1064386"}

b. Provision Crawl Component

Create a crawl component for new crawl topology. $ct can be passed as the identified crawl topology. $si can be passed as the search instance. We still need to know the crawl store ID because it's required via -crawldatabase parameter to create the new-spenterprisesearchcrawlcomponent.

To find the Guid we can run the following:

$searchapp.crawlstores

Copy the GUID for ID. For Example:

74278909-cb14-46ee-96e2-69ad77ffdfc0

Create a new crawl component.

new-spenterprisesearchcrawlcomponent -searchapplication $searchapp -crawltopology $ct -crawldatabase "74278909-cb14-46ee-96e2-69ad77ffdfc0" -searchserviceinstance $si

Finally, set the new crawl topology as active.

$ct | set-spenterprisesearchcrawltopology -active

c. Create Query Topology

Create new Query topology into object $qt

$qt = $searchapp | new-spenterprisesearchquerytopology -partitions 1

$p1 = ($qt | get-spenterprisesearchindexpartition)

d. Provision query component

Create a query component for new query topology

new-spenterprisesearchquerycomponent -indexpartition $p1 -querytopology $qt -searchserviceinstance $si

$p1 | set-spenterprisesearchindexpartition

Assign the property store to the index partition

$p1 | set-spenterprisesearchindexpartition -propertydatabase "GUID HERE"

Where is the propertydatabase GUID?

$searchapp.propertystores to copy the ID "GUID"

Finally, set the query topology as active

$qt | set-spenterprisesearchquerytopology –active

SharePoint 2010 Multi-Tenant Hosting Part 2 “Configuring”

2009-10-20T18:24:25Z

Part 2 - Configuring Hosting

Configuring hosting requires powershell so the steps are all based off of using it.

1. Create a subscription and assign sites to it:

$sub = new-spsitesubscription

$sub

2. Pulling the site collection or set of site collections you wish to join to the site group:

get-spsite

$site = get-spsite | where {$_.url -eq "http://contoso"}

$site

Now you have two variables. Variable 1 $sub object contains a new spsitesubscription. Variable 2 $site contains a site collection.

3. Add the site collection $site, to the newly created site subscription $sub.

set-spsite -identity $site -sitesubscription $sub

Check whether it has been added correctly by doing the following:

get-spsitesubscription

If a database ID exists, then you can type the following

get-spdatabase | where-object {$_.id -match "full or partial guid"}

Will output the results of the associated site collection.

4. Create a secondary subscription and associate a different site collection within same web application for demonstration purposes using the steps above.

5. Create a SubscriptionSettings Service Application and Proxy

A. Start the WSS Subscription Settings Service

B. Create Service Application and Proxy via PowerShell

$appPool = New-SPIISWebServiceApplicationPool -Name SettingsServiceApppool -Account domain\use

$sa = new-spsubscriptionsettingsserviceapplication –Name SubscriptionSettingsServiceApplication –Databasename SubscriptionSettingsServiceApplicationDB –applicationpool $appPool

$sap = new-SPSubscriptionSettingsSericeApplicationProxy –ServiceApplication $sa

6. Creating the Tenant Admin Site for each site group

$sub = get-spsitesubscription –identity “http://server”

$tasite = new-spsite –url “http://Contoso/sites/tasite1” –template “tenantadmin#0” –owneralias domain\username –sitesubscription $sub

7. Provision a search service application in hosting mode. Please see the Configure Search Service application using Powershell blog.

8. Feature's and hosting

Once a feature has been installed into the farm, it's available to all sites and can be activated through manage features pages. The way to control this in a hosting scenario is you only provide the features available to a given site group through various PowerShell commands. Any features not listed are excluded and not available for all site collections that belong to the corresponding site group. Steps are below:

Create a feature set:

$fs =New-SPFeatureSet

Adding features to a feature set:

$farm = Get-SPFarm

$feature1 =$farm.FeatureDefinitions | where{$_.ID -eq "02464c6a-9d07-4f30-ba04-e9035cf54392"}

Add-SPFeatureSetMember -Identity $fs -FeatureDefinition $feature1

Adding feature set to subscription:

Set-SPSiteSubscriptionConfig -Identity $sub -FeatureSet $fs

SharePoint 2010 Multi-Tenant Hosting Part 1

2009-10-20T18:19:05Z

Hosting 101 Part 1

Before providing the steps of setting up hosting, it's important to understand the main concepts behind O14 hosting.

O14 has a rich set of hosting features which triumph over the previous version in many ways. It's now simple to setup hosting on the site collection level.

For Example:

Tenant "customer 1" on site collection 1 is hosted on web application A
Tenant "customer 2" on site collection 2 is hosted on web application A

Each tenant would only have full administrator rights on his/her assigned site collection. Customer 2 and users accessing site collection 2 wouldn't be able to access Customer 1's site.

Also, service applications that are in hosting mode would keep each tenants data separate from another tenants. For example, one shared search service application could service customer 1\site collection 1's data and customer 2/site collection 2's data while keeping them separate from each other.

For Example: Users searching in site collection 1 will not be able to search and find content that resides in site collection 2. Users searching in site collection 2 will not be able to search and find content that resides in site collection 1. They will be able to search and locate data within the site collection they are searching from.

Site Groups

The segmentation is possible through the use of site groups also known as site subscriptions. In the example above, customer 1's site collection belongs to site group 1 and customer 2's site collection belongs to site group 2.

Things to know about site groups:

1. Sites can belong to only one site group at a time.

2. Sites cannot join a site group that contains sites that exists on a different web application.

3. A site group can span across more than one content database

4. No GUI interface for managing site groups. PowerShell is required to create/manage/remove site groups.

Tenant Admin Site

A hosted customer is referred to as a tenant. You can provision a tenant admin site which gives the tenant full administrator rights over the site collection. The tenant admin site can be used to create additional sites for example after self service site creation is enabled.

Service Applications and Hosting

Be default, a service application is consumed at the web application level. So all sites, under a web application would consume from the same service application and data would be shared. In hosting mode, a shared service application partitions data where every site group has its own partition. The partition isn't shared meaning other site groups wouldn't be able to see this data even though all sitegroups are using the same service application. Configuring hosted service applications may differ based on the type of service application that is being deployed.

For Example: Deploying a Shared search service application requires you to use PowerShell with addition of -partitioned switch

Features

It's also possible to deploy features in an "a la carte" manner to site groups. So one site group can have more features available than another. This is configurable from PowerShell.

Hosting Part 2 provides a step by step walk though using PowerShell

Configuring Kerberos Authentication in SharePoint 2010 Part 1

2009-10-20T18:14:00Z

Configure Kerberos Authentication in O14 Part 1

When configuring Kerberos with O14 you will be using IIS 7.0.

Integrated windows authentication is now handled in kernel mode and enabled by default. This technically was primarily for ease of use and performance boosts since auth is no longer happening in user mode.

What are the benefits?

No longer need to register SPN’s for application pool accounts because the AD computer account will be used to decrypt the service ticket. This is true even if the application pool identify is domain\username.

Note: This works as long as you’re using the netbios or FQDN of the server to access a site.

Unfortunately, in a Web Farm scenario like SharePoint this doesn’t work because you won’t be passing the Netbios/FQDN of the server to access the site unless it’s a standalone install for testing purposes.

The work around for enabling Kerberos in a web “SharePoint” farm you need to specify the application pool identity for the associated web application. Then you need to create an SPN using setspn tool.

Farm scenario task:

Enable Kerberos for the following:

· SharePoint Web Application “Sharepoint – 80”

· Site is named contoso.com

· Web Application is using domain account, ”contoso\farmadmin”, as application pool identity.

Step 1: Setting useAppPoolCredentials to true in application.config file.

The goal is to continue to use kernel mode authentication in IIS 7.0. I don’t recommend disabling it for the pure purpose of using Kerberos in an O14 environment. The first step is setting the useAppPool Credentials to true in application.config file for the associated web site.

In this example, I want to set this attribute on my “SharePoint – 80” web application:

Locate the application.config file in the following dir:

c:\windows\system32\inetsrv\config\

When you open application.config host file with notepad you will see something like this for web application “WebApplicationName”:

<system.webServer> <security> <authentication> <windowsAuthentication enabled="true" </authentication> </security> </system.webServer>

I don’t like the fact that useKernelMode was missing from the application.config file by default. The following steps walk you through enabling both attributes.

A.) Disable Kernel-mode authentication for the associated web site by going into IIS Manager. Select the website\authentication\Advanced Settings and uncheck Enable Kernel-mode authentication.

B.) Perform IIS reset and then go back into Advanced settings and re-enable Kernel-mode authentication by checking the check box. Perform an IIS reset.

C.) Launch the applicationhost.config file within c:\Windows\System32\inetsrv\config to verify you see the following:

<system.webServer> <security> <authentication> <windowsAuthentication enabled="true" useKernelMode="true"/> </authentication> </security> </system.webServer>

D.) Run appcmd and set useAppPoolCredentials attribute to true for the associated web application.

Appcmd set config “SharePoint - 80” /section:windowsauthentication /useAppPoolCredentials:true /commit:MACHINE/WEBROOT/APPHOST

Now checking application.host config file you should see the following for the associated web application:

<system.webServer> <security> <authentication> <windowsAuthentication enabled="true" useKernelMode="true" useAppPoolCredentials="true" /> </authentication> </security> </system.webServer>

Step 2: Trust the service account for delegation

A.) Launch Active Directory Users and Computers

B.) Locate account running as the application pool identity

C.) Go to properties on the account, select delegation tab

D.) Select “Trust this user for delegation to any service (Kerberos only)

Step 3: Set SPN

SPN is required to map the service/host name to the Application Pool identity.

A.) Install SPN from the following location:

http://www.microsoft.com/downloads/details.aspx?familyid=5fd831fd-ab77-46a3-9cfe-ff01d29e5c46&displaylang=en

B.) From cmd prompt, run the following cmd:

Setspn.exe –a http/contoso.com contoso\farmadmin

Step 4: Enable Kerberos on the Web Application

A.) Launch Central Admin and select Application Management

B.) Select Manage Web Application and choose the appropriate web application

C.) From the ribbon, select Auth Providers

D.) Select the associated zone and enable Negotiate (Kerberos) and save

Step 5: Verify that Kerberos authentication is working

A.) Go to the security log on the WFE

B.) Filter on all Event ID’s 4624’s

In the above event, you can see the logon process is using Kerberos. If you scroll up on the event further, you can also get the source computer as well as user account used to log in. It’s easier to filter on this event with the logon account to confirm on a high traffic server.

With part two, I’ll discuss using Kerberos with service applications.

Where have I been?

2009-10-15T18:45:44Z

I’m checking in as it’s been awhile since I’ve posted anything. I’m currently a beta rotational engineer supporting SharePoint 2010. I’ve been so wrapped up in supporting the new product in it’s early stages that I haven’t had a lot of time to post on 2007. I’ll be providing an ample amount of blog post on SharePoint 2010 so feel free to check back at a later date.

Next week, I’ll be attending the SharePoint conference Oct 19-22nd in Vegas and will be available via the CSS booth in the exhibition hall. Stop by and say hello and feel free to ask questions.

Configuring Performance Monitor Log for SharePoint Performance Issues.

2009-05-28T04:12:00Z

This is part 2 of troubleshooting SharePoint performance problems. In order to properly diagnose and troubleshoot SharePoint performance problems, Microsoft CSS support often depends on a performance monitor (perfmon) log during problem period. In addition, we might request a baseline perfmon of healthy behavior.

Part 2 of this blog is dedicated to walking you through setting up performance monitor. Why is that necessary? We have a very specific method in collecting this data. A shout out is in store for Jason at MSFT for originally documenting most of these steps.

Steps are below:

1. Open Performance Monitor, click the + button. Under Performance object, select .NET CLR Memory. Under "Select counters from list", verify that you see Process ID

In this case, it's clear that the PID information is missing so you must fix the counters as this is what enables a mapping of w3wp to a process ID.

a) Click Start, click Run, type cmd, and then click OK.

b) At the command prompt, type unlodctr .NetFramework, and then press ENTER.

c) Use the cd command to change to the Windows\Microsoft.Net\Framework\v2.0.50727 folder.

d) At the command prompt, type lodctr corperfmonsymbols.ini, and then press ENTER

Reference the following article for more information: http://support.microsoft.com/kb/922775

2. Add this registry key so that Performance Monitor will list the PID of the W3WP as part of the counter detail.

The following KB article explains how to do this: http://support.microsoft.com/kb/281884

3. Open Performance Monitor

4. Expand Performance Logs and Alerts

5. Right click on Counter Logs, click New Log Settings and type a name for the Perfmon log and click OK

6. Click on the Add Objects button. Then add the following objects and click Close

.NET CLR Exceptions
.NET CLR Loading
.NET CLR Memory
.NET Networking
ASP.NET
ASP.NET Applications
Memory
Network Interface
Physical Disk
Process
Processor
System
Web Service
Web Service Cache

Note: Add the Threads object when troubleshooting CPU Spikes/CPU high utilization.

7. Make sure to use "Run As" so we can accurately capture the .NET counters. This should be run as the same account as the application pool.

8. Setup performance logging at 5 second intervals.

9. Select the Log Files tab and Configure button and set the preferred directory path and file name. Limit the log file size to 250 MB and hit OK.

Example screenshot below:

10. Select the Schedule tab and the start up option is your decision on whether or not you want to automatically start the log or manually.

The Stop Log section settings need to match the following screenshot:

The goal is that no information is going to be overwritten. So when the buffer limit of 250 MB is reached, a new perfmon log will be created and the existing one is preserved.

Hit okay and if you set it to start automatically based on a specified time, then you can check Performance Monitor and verify its running. If you set the Start Log option to manual, then you need to go into Performance Monitor and manually start the log.

Great, now perfmon log is up and running but we still need one more piece of information. The PID information is helpful but we need to be able to map which PID or PID's are mapped to a specific application pool. This requires the use of IISAPP. Steps for running IISAPP are the following:

11. Go to the same server you configured perfmon and launch cmd prompt. Type the following:

iisapp

After hitting enter, you might get prompted to register cscript. Hit yes and run iisapp again.

You will see an output like the following:

You can see the value in the screen shot. Now when I launch a perfmon to review and look at process information, I can map the PID to the associated application pool.

For data collection purposes, I recommend piping this out to a txt file like running iisapp > c:\iisappout.txt.

I would run this at least every 30 minutes and you can easily configure a script or batch file to automate this process.

Note: IISAPP doesn't work in Windows 2008. In order get PID <--> Application Pool mapping you must run the following:

Launch cmd prompt and run the following command from the c:\Windows\System32\inetsrv directory:

appcmd.exe list wp

Question/Answer Section

Q: Why do I suggest 5 second interval for sampling period?

A: It really depends on how often the performance problem reproduces and how long you decide to leave the log running. If your capture period is 8 hours or less, then I recommend 5 seconds. If your capture period is more than 8 hours, I assume that it's unknown when the problem will occur which requires a long running perfmon. In this case, I recommend a sample period of 120 second interval. The key is managing how rapidly the perfmon log (BLG file) grows in size.

Q: Shouldn't I set the log to overwrite instead of creating new BLG's during long running perfmon sample?

A: This is personal preference but several conditions exists where you probably want to set the log to overwrite

Conditions:

1. You are limited on available free space on directory holding the perfmon output (BLG files)

2. Your not sure when the performance problem will reproduce and it could take 2/3 days or even a few weeks.

I will cover a nice in depth look of SPDiag in part 3 of Troubleshooting SharePoint Performance series.

Troubleshooting Security Only Crawl

2009-02-09T05:19:00Z

What is Security Only Crawl?

Before troubleshooting Security Only Crawl’s, let’s start out with what is a security only crawl? Security Only crawl's take place when users are added/removed from Sharepoint groups and/or explicitly added/removed from a list. When incremental crawl starts, these security changes, “Updated ACL’s”, must be pushed down to all affected items within the index. We security trim these results.

For Example, User A is able to search on all items in a specific document library named HR Docs. You don’t want User A to be able to search and discover documents which were crawled previously from Document Library named HR Docs. You remove User A from the only Sharepoint group that has permissions to view these documents and iniate an incremental crawl. What will happen is that the security change will be discovered and the updated ACL will be processed. The gatherer will commit the security changes to all items within HR Docs library. When the security only crawl is completed, User A will not return results for querying items within HR Docs library.

Note: The items within the document library are not fetched but the security of the item is.

Troubleshooting Security only crawls?

This is an interesting topic and one that is rarely documented which is the primary reason for this blog. What would necessarily lead you down the path to troubleshooting security only crawls? Well there are a few scenarios such as:

1.) User A is still able to search and return items which he shouldn’t be due to being removed from Sharepoint groups.

2.) User A is not able to search items even though he was recently added to a Sharepoint group with rights to the items in question.

3.) Incremental crawls are intermittently taking a long time to crawl very few items.

Scenario three is more frequent and tends to cause the most confusion for Sharepoint Administrators so I’ll discuss this in more detail. What you will learn from scenario three should help you in troubleshooting scenarios one and two.

Scenario 3: My incremental crawls are taking longer than expected to crawl a very small # of items.

You perform an incremental crawl which takes 30 minutes. You perform an incremental crawl the following day and crawl the same # of items and it takes 3 hours. This starts to become a trend and typically raises a red flag. The other symptom is that the crawl status page within Central Administrator will appear as crawling with no items updated for a long period of time. Eventually, the crawl status will update with crawl complete. The assumption is that the crawler appears to be getting hung or stuck during these long incremental crawls. The problem is that you really don’t know if the crawler is still running or not. I typically see the above symptoms when security only crawls are taking place. Please keep in mind their are several moving parts to the incremental crawl process and I'm just identifying a common one.

Note: There is no easy way to tell if a security only crawl is running. The reason is because the search status page doesn’t log any security related events. The only thing you will see is that it’s still crawling.

How can you verify that security only crawls are running?

I found two methods to assists in determining if security only crawls are occurring.

Method 1:

Whenever a document is added/removed or User is added/removed from a group, this change gets logged in the change log “eventcache table” within that associated content database. Please see my original blog in determining which content database you should be checking the change log post incremental crawl period. The key is comparing the change log between short/long incremental crawls to see what’s different between the two.

The eventtypes column within Eventcache table contains the type of changes that the crawler will enumerate through during incremental crawl time.
This is one way to determine if security only crawls are contributing to your long crawl times. You will need to query the eventtype column for all events that were part of your long running incremental crawl.

You would be looking for event types 4194304 and 2097152.

2097152 - “A user is added to a Sharepoint Group”
4194304 - “A user is deleted from a Sharepoint Group”

Steps are the following:

A.) Dump the MSSCrawlHistory from Search database table and look for the latest Crawl's with Status 11 (One long running and one that ran quickly)

B.) The screen shot shows me Crawl ID 8162 and 8165 "I'll use these crawl ID's in my example"

C.) Next, open the MSSChangeLogCookies table within the Search DB and you'll see something like the following for the corresponding crawl ID's gathered from reviewing the MSSCrawlHistory table:

ChangeLogCookie_old will contain:

1;0;888eef75-d584-4edf-b242-f5161d4c3c44;633686171531600000;2564

ChangeLogCookie_new will contain:

1;0;888eef75-d584-4edf-b242-f5161d4c3c44;633686187284500000;2577

It’s important to understand the bold values above. The long Guid “888eef75-d584-4edf-b242-f5161d4c3c44” is the actual database were crawling against. This is important because this is the content database where you’ll need to query the eventcache table for the batch of changes processed during this crawl. To find the actual database value you can run something like the following against Config DB:

select * from objects with (NOLOCK) where ID = '888eef75-d584-4edf-b242-f5161d4c3c44'

The last 4 digits 2564 and 2577 are the changes from the eventcache table that the crawler processed.

D.) Next, You want to run a query against the eventcache table and output to a file:

Select * from eventcache with (NOLOCK) where ID BETWEEN 2564 AND 2577

E.) Perform the above steps for both short/long running incremental crawls and compare the differences from the eventcache output via Step D.

Conclusion from Method 1: If you see eventtypes 4194304 and 2097152 only during your long running incremental crawls, it’s a fairly safe bet to say that the security only crawls are the issue. I would combine both methods to confirm that 99 %.

Method 2:

When the crawl log stops reporting events and appears hung again you should open the ULS logs on the indexer and see a ton of events like the following:

04:21.0 mssearch.exe (0x092C) 0x09A8 Search Server Common GathererSql 0 Medium CGatherer::CommitTransaction succeeded, URL sts3://site/siteurl=/siteid={ee7f629c-a36c-4327-b5f1-0652878fa4a3}/weburl=docs/webid={ba2b56ba-27d1-4cfb-90b4-e638d3f3b0ed}/listid={4d587236-3886-440f-a0d7-0d33622724cd}/folderurl=/itemid=27, CrawlID 51, SourceDocID 229 - File:d:\office\source\search\search\gather\server\gatherobj.cxx Line:9449

This event tells us that the gatherer is busy committing transactions to the index. You’ll notice some subtle differences between the events. For Example: Itemid and SourceDocID will be different for every event.

The goal is finding out the following about the event:

1.) What exactly is being committed to the index?
2.) Was this an item that was actually added/changed?

To figure out which item is being hit you would run the following SQL queries based on the SourceDocID, List ID, and ItemID. I use the above event as an example:

Query 1 against SearchDB:

select * from MSSCrawlURL where docID = '229'

"Look at the DisplayURL and see what URL it's hitting as this usually includes a file name"

Query 2 against ContentDB:

select * from AllDocs where listid = '4d587236-3886-440f-a0d7-0d33622724cd' and doclibrowid = '27'

“Look at the LeafName and confirm this matches the same item as Query 1”

Both of these queries should result in the same file. I would run the above two queries on multiple events. Maybe grab like 10/20 items and then hit the Crawl logs after the crawl completed and see if these items were crawled. If not, then it’s very safe to say that security only crawls were taking place in addition to the method I provided earlier.

Question:

Why do security only crawls take so long?

Answer:

The time difference in crawl can be attributed to expansion of the SharePoint Group and also that the group is at the site collection level and affects items beyond the list. If a Sharepoint group has several thousand users at site collection level, you can see how this can be very expensive. Also, a large number of items within that site collection can add to the delay because new ACL changes will be pushed down to every item affected by the security change.

Question:

How can I work around this and prevent security only crawls from affecting incremental crawl times?

Answer:

Instead of users explicitly added to Sharepoint groups, add AD groups instead. Managing adding/removing users from Active Directory security groups will not cause ACL changes within Sharepoint. Because of this, no security only crawls will occur.