SharePoint Brew

Why SharePoint 2013 Cumulative Update takes 5 hours to install?

2013-04-02T02:58:37Z

I recently went through the process of applying the latest SharePoint 2013 Cumulative Update on my multi server SharePoint 2013 Farm. To my surprise, I found the total time to apply the patch (install the bits) took over 5 hours per SharePoint Server. This time doesn’t include running PSConfig which is considered phase 2. My SharePoint 2013 farm is hosted by a Windows 8 Hyper V environment where my VM’s meet the minimum hardware requirements for memory etc. I found that as I started to gracefully terminate other SharePoint services running, the time of subsequent patch runs started to drop tremendously. I suspect the following:

Some of these processes like App Fabric, Search(node runner), etc.. are taking enough CPU time, threads associated with Windows Installer gets put into a longer line waiting for the most popular roller coaster ride which happens to be the CPU. Note: Threads are treated the same as everyone else in terms of thread priority.

Another way of saying this:

By having x # of additional processes running, you increase the # of threads vying for CPU time by x much. Assuming each process threads are at the same priority, each thread will evenly get a slide of Processor time to execute.

For Example: Let’s assume I have 2 processes (process A) and (process B). Process A has 10 run-able threads while Process B has 3 run-able threads. Assuming these 13 threads are at the same priority, each thread would theoretically receive one-thirteenth of CPU time.

I wrote a Power Shell script to automate and speed up the install of a Cumulative Update on SharePoint 2013. This Power Shell script improved my patch time from 5 hours to 16 minutes. The patch performs the following steps:

1. Disable the IISAdmin, App Fabric, and SPTimerV4 service

2. Shut down IIS Admin, App Fabric, and Timer Services if they are running

3. Give you the option to Pause the Search Service Application (see search notes below)

4. Stop Search Services (see search notes below)

5. Install the patch in passive mode (No user interaction required but will witness the patch install in the UI)

Note: Power Shell should remain open in the background while patch is running

6. Upon completion of the patch, the Power Shell script, services in step 1 are set to Automatic

7. Starts up IIS Admin, App Fabric, and Timer Services

8. Starts up Search Services

9. Resume the Search Service Application if it was paused

10. Finally, the script will display the Start Time and End Time for patch install

Search Notes:

This script will only stop Search SharePoint Services if they are running

This script will only pause the Search Service Application upon user input

This script will only start Search SharePoint Services if they were stopped

This script will only resume the Search Service Application if it was paused

Important: I recommend pausing the Search Service Application for several reasons but primarily so that Search services aren’t being shutdown during a Full, Incremental, or Continuous Crawl. I made this an option because a SharePoint Admin can run the patch simultaneously on multiple SharePoint servers and may prefer to manually pause/resume the Search Service Application.

Instructions for Running the Power Shell Script

In order to run the script, save the flow script to a text editor like notepad. Save the file with a .ps1 extension to a folder on the SharePoint Server. Drop the SharePoint cumulative update in the same folder as the script. In my example, I created a scripts folder where I dropped the powershell script and the cumulative update.

For Example:

Important: Only one update should be placed here and the file extension should be .exe.

Finally, to run the script from c:\scripts using SharePoint Management Shell. This script is tested on Windows 2012 running SharePoint 2013.

Example Screenshot – during patching it looks like:

Example Screenshot – Patching is complete!

Updated Script 4/4 1:41 AM (CST). Thanks to Scott Manning for script validation efforts with appfabric.

Updated Script 4/2 2:10 PM (CST). Thanks to Jon Waite for script validation efforts.

Script is below, start copying and enjoy!

<# ==============================================================
//
// Microsoft provides programming examples for illustration only,
// without warranty either expressed or implied, including, but not
// limited to, the implied warranties of merchantability and/or
// fitness for a particular purpose.
//
// This sample assumes that you are familiar with the programming
// language being demonstrated and the tools used to create and debug
// procedures. Microsoft support professionals can help explain the
// functionality of a particular procedure, but they will not modify
// these examples to provide added functionality or construct
// procedures to meet your specific needs. If you have limited
// programming experience, you may want to contact a Microsoft
// Certified Partner or the Microsoft fee-based consulting line at
// (800) 936-5200 .
//
// For more information about Microsoft Certified Partners, please
// visit the following Microsoft Web site:
// https://partner.microsoft.com/global/30000104
//
// Author: Russ Maxwell (russmax@microsoft.com)
//
// ---------------------------------------------------------- #>

###########################
##Ensure Patch is Present##
###########################
$patchfile = Get-ChildItem | where{$_.Extension -eq ".exe"}
if($patchfile -eq $null)
{
Write-Host "Unable to retrieve the file. Exiting Script" -ForegroundColor Red
Return
}

########################
##Stop Search Services##
########################
##Checking Search services##
$srchctr = 1
$srch4srvctr = 1
$srch5srvctr = 1

$srv4 = get-service "OSearch15"
$srv5 = get-service "SPSearchHostController"

If(($srv4.status -eq "Running") -or ($srv5.status-eq "Running"))
{
    Write-Host "Choose 1 to Pause Search Service Application" -ForegroundColor Cyan
    Write-Host "Choose 2 to leave Search Service Application running" -ForegroundColor Cyan
    $searchappresult = Read-Host "Press 1 or 2 and hit enter"
    Write-Host
    if($searchappresult -eq 1)
    {
        $srchctr = 2
        Write-Host "Pausing the Search Service Application" -foregroundcolor yellow
        Write-Host "This could take a few minutes" -ForegroundColor Yellow
        $ssa = get-spenterprisesearchserviceapplication
        $ssa.pause()
    }
    elseif($searchappresult -eq 2)
    {
        Write-Host "Continuing without pausing the Search Service Application"
    }
    else
    {
         Write-Host "Run the script again and choose option 1 or 2" -ForegroundColor Red
        Write-Host "Exiting Script" -ForegroundColor Red
        Return
    }
}

Write-Host "Stopping Search Services if they are running" -foregroundcolor yellow
if($srv4.status -eq "Running")
{
    $srch4srvctr = 2
    set-service -Name "OSearch15" -startuptype Disabled
    $srv4.stop()
}

if($srv5.status -eq "Running")
{
    $srch5srvctr = 2
    Set-service "SPSearchHostController" -startuptype Disabled
    $srv5.stop()
}

do
{
    $srv6 = get-service "SPSearchHostController"
    if($srv6.status -eq "Stopped")
    {
        $yes = 1
    }
    Start-Sleep -seconds 10
}
until ($yes -eq 1)

Write-Host "Search Services are stopped" -foregroundcolor Green
Write-Host

#######################
##Stop Other Services##
#######################
Set-Service -Name "IISADMIN" -startuptype Disabled
Set-Service -Name "appfabriccachingservice" -startuptype Disabled
Set-Service -Name "SPTimerV4" -startuptype Disabled
Write-Host "Gracefully stopping IIS W3WP Processes" -foregroundcolor yellow
Write-Host
iisreset -stop -noforce
Write-Host "Stopping Services" -foregroundcolor yellow
Write-Host

##Gracefully stopping Distributed Cache##
$srv1 = get-service "appfabriccachingservice"
if($srv1.status -eq "Running")
{
    Write-Host "Gracefully stopping Distributed Cache, this could take a few minutes" -foregroundcolor Yellow
    Stop-SPDistributedCacheServiceInstance -Graceful
    Write-Host "Distributed Cache disabled"
    Write-Host
}

$srv2 = get-service "SPTimerV4"
if($srv2.status -eq "Running")
{$srv2.stop()}

Write-Host "Services are Stopped" -ForegroundColor Green
Write-Host
Write-Host

##################
##Start patching##
##################
Write-Host "Patching now keep this PowerShell window open" -ForegroundColor Magenta
Write-Host
$starttime = Get-Date

$filename = $patchfile.basename
. .\$filename /passive

Start-Sleep -seconds 20
$proc = get-process $filename
$proc.WaitForExit()

$finishtime = get-date
Write-Host
Write-Host "Patch installation complete" -foregroundcolor green
Write-Host

##################
##Start Services##
##################
Write-Host "Starting Services Backup" -foregroundcolor yellow
Set-Service -Name "appfabriccachingservice" -startuptype Automatic
Set-Service -Name "SPTimerV4" -startuptype Automatic
Set-Service -Name "IISADMIN" -startuptype Automatic

##Grabbing local server and starting appfabric service##
$servername = hostname
$server = get-spserver $servername
$dcache = Get-SPServiceInstance | where{($_.TypeName -eq "Distributed Cache") -and ($_.Server -eq $server)}
if($dcache.status -eq "Disabled")
{
Write-Host "Starting Distributed Cache Service" -foregroundcolor "Yellow"
$dcache.start()
}

$srv2 = get-service "SPTimerV4"
$srv2.start()
$srv3 = get-service "IISADMIN"
$srv3.start()
$srv4 = get-service "OSearch15"
$srv5 = get-service "SPSearchHostController"

###Ensuring Search Services were stopped by script before Starting"
if($srch4srvctr -eq 2)
{
    set-service -Name "OSearch15" -startuptype Automatic
    $srv4.start()
}
if($srch5srvctr -eq 2)
{
    Set-service "SPSearchHostController" -startuptype Automatic
    $srv5.start()
}

###Resuming Search Service Application if paused###
if($srchctr -eq 2)
{
    Write-Host "Resuming the Search Service Application" -foregroundcolor yellow
    $ssa = get-spenterprisesearchserviceapplication
    $ssa.resume()
}

Write-Host "Services are Started" -foregroundcolor green
Write-Host
Write-Host
Write-Host "Script Duration" -foregroundcolor yellow
Write-Host "Started: " $starttime -foregroundcolor yellow
Write-Host "Finished: " $finishtime -foregroundcolor yellow
Write-Host "Script Complete"

Thanks,

Russ Maxwell, MSFT

SharePoint 2013 Search – Updating the Refinement Panel

2013-03-15T18:16:26Z

This is my first of many SharePoint 2013 blogs and I’m starting with Search. I like the new Search UI for users and the look is so much cleaner with the hover panel, search blocks, etc.. I’m really curious how customizing the Search refinement panel works in SharePoint 2013. I was surprised to see that Company was missing from the refinement panel on the left when I initiated a query and returned items.

For Example:

In my test scenario, I added several different types of office documents where the properties of those documents had a specific Company populated. I want to add Company to the refinement panel in 2013 Search so that I can drill down the search results based on Company. For those that don’t know, the refinement panel is a way to drill down on the base search results by only returning results based on a specific property.

For Example: I query weather and only want to retrieve results authored by Ron Burgundy so I click on Ron Burgundy from the refinement panel on the left.

Managed Property vs Crawled Properties and the Search Schema

In order to start, you must understand the concept of Managed Properties verse Crawl Properties. This basic concept hasn’t changed from SharePoint 2010. As you crawl items, those (properties\property values) associated with that item are crawled as well and present in the Search Schema as crawled properties. The crawled properties by themselves can’t be queried. That’s where Managed Properties come in, you create managed properties and map those to crawled properties by accessing the Search Schema. Subsequently, users can query the managed properties with a specific query and return results. The Search Schema consists of Managed Properties/Crawled properties and defines what attributes are available for a given property. The search schema controls what properties can be searched and controls how they are presented within Search enabled sites. Modifying the Search Schema is a fancy way of saying I want to add new managed properties and map them to crawled properties in the Search Admin UI or Power Shell.

Modify the Search Schema (Add Managed Property and Map to Crawled Property)

In order to start, we must modify the Search Schema in order to add a managed property. It’s now possible to update the search schema within the context of a Site Collection but this blog will be focused on how to do this at the Search Service Application Level so that it will be pushed down to all Web Applications consuming from it. So again, these steps will walk you through adding company to the refinement panel. The first steps are to Create a Managed Property and Map it to the Company crawled property.

Steps are the following:

1. Access Central Administrator and Under Application Management select Manage service applications

2. Select the Search Service Application, and hit the Manage button from the Ribbon

3. Next, Select Search Schema link on the left under Queries and Results section

4. Ensure you select the Managed Properties Link and in the Managed Property field, type orgnames and hit enter

5. Select the drop down next to OrgNames and choose Edit/Map Property

6. Scroll down and under Alias, replace Company with “NotUsedCompany” and scroll down and hit OK

7. Hitting OK should bring you back to the Search Schema Page and click on the New Managed Property link

8. For the Name I typed CustomCompany and dropped a description in and the is specified as Text

9. At a minimum check the following options: (Queryable, Refinable (yes))

10. Optionally, I also checked Allow Multiple Values and Sortable

11. Scrolling down I now entered Company as the Alias and also left Token Normalization checked

12. Now, the most important part is add a mapping to a crawled property so in this case I click on “Add a Mapping”

13. Within Crawled property selection choose All categories and type company and hit Find

14. Select Company and hit OK and it should now look like:

15. Scroll down leaving the remaining options as default and click OK

16. This takes you back tot he Search Schema so enter customcompany in the Managed Property field and ensure it looks like this:

17. Now perform a full crawl on your desired content source by clicking on the Content Sources link on the left

18. Now hit the pull down next to the desired content source and choose Full Crawl

19. After completion move to the next steps

Modify the Search Results page and update the the Refinement Web Part

After the full crawl has completed, The final step is to update the refinement web part on the Search Results page for a Search Site. The steps are the following:

1. Access your Search Site and perform a simple search to get to the results.aspx page.

Note: Notice my results have the standard refinement options like Result Type, Author, and Modified Date

2. Next hit the gear on the top right of the page and select Edit Page

3. On the left, hit the pull down next to the Refinement web part and click “Edit Web Part”

4. The Properties for the Refinement web part will be on the right so select the Choose Refiners… button:

5. On the list of Available refiners on the left, choose the CustomCompany and click Add. Under Configuration For: Custom Company window, update the Display Name to Company. It should look like this:

Note: On the right, I hit Move Up to place CustomCompany above Last Modified Time

6. Hit OK and that should take you back to the Search Results page where you can perform additional Edits. Hit OK to the Refinement web Part properties dialog on the right

7. Check In your changes by hitting the Check In button from the Ribbon and finally Publish

8. Now perform a search query and validate Company is now available in the refinement panel

Question: Why did you update the display name of the orgnames managed property?

Answer: Each display name is unique so you can’t have the same alias name for two or more managed properties.

Question: I followed your steps and I don’t see Company from the refinement panel after performing a search?

Answer: The most likely cause is that you need to index or reindex content that has values populated for Company property. Finally, ensure your searching for terms contained in that document.

Thanks!

Russ Maxwell, MSFT

SharePoint Search 2010: Crawl status returns 503 Server Unavailable

2013-03-12T03:05:31Z

I ran into an interesting problem where accessing the 2010 Search Service Application administration page returns a 503 Server unavailable for the Crawl Status.

Usually, a 503 indicates that the application pool is stopped that is hosting a site. In this case, it’s the SharePoint Web Services Virtual directory hosting the Search Admin web service. I selected the virtual directory, right clicked, and chose Advanced Settings and the GUID for the application pool is present.

Accessing the Application Pool shows that it is indeed stopped.

Cause 1

Try starting the application pool and refreshing the page. If it still fails and the application pool is stopped again proceed to cause 2.

Cause 2

Usually, this problem occurs because the application pool identity doesn’t have appropriate permissions in SQL. For the service account running Search Service Application, you’ll need to ensure this account has the following roles on SQL Server.

db_owner and public roles on the following:

Search Crawl Store Database
Search Service Application Database
Search Property Store Database

WSS_Content_Application_Pools role on the following

SharePoint_AdminContentDatabase
SharePoint_Config Database

Cause 3

The Search Service Account is missing from the local WSS_WPG group. Access computer management, local users and groups, and ensure the Search Service Account is a member of this group.

Cause 4

This is a rare scenario but entirely possible. If you check the ULS logs you may see something like the following when rendering the Search Admin page:

03/11/2013 21:46:10.42 w3wp.exe (0x1694) 0x08DC SharePoint Server Search Administration dn26 High System.Net.WebException: The remote server returned an error: (503) Server Unavailable. at Microsoft.Office.Server.Search.Administration.SearchApi.RunOnServer[T](CodeToRun`1 remoteCode, CodeToRun`1 localCode, Boolean useCurrentSecurityContext, Int32 versionIn) at Microsoft.Office.Server.Search.Administration.SearchApi..ctor(SearchServiceApplication application) at Microsoft.Office.Server.Search.Administration.SearchServiceApplication.get_SearchApi() at Microsoft.Office.Server.Search.WebControls.SearchApplicationSystemStatus.GetStatusString(SearchServiceApplication searchApp) at Microsoft.Office.Server.Search.WebControls.SearchApplicationSystemStatus.BindContentToTable(Int32 pageNumber, Table contentTable) 208aa6ae-52e4-48d6-8168-414f56bb56fd

This can happen if the IIS_IUSRS group is missing from the local security policy. Specifically, the group needs to be added to the following policies:

Impersonate a client after authentication and Log on as a batch job

Check and Fix:

Go to Start, All Programs, Administrative Tools, and select Local Security Policy.
Expand Local Policies, User Rights Assignment and check Log on as a batch job and Impersonate a client after authentication
If IIS_IUSRS group is missing proceed to step 4. If it’s present this is not the issue and disregard.
Right click each policy go to Properties and add IIS_IUSRS
Reboot or Perform a GP Update

Note: If your unable to edit the local policy, you’ll need to contact your AD Administrators to update the policy.

It should look like the following after completion:

Resources:

http://technet.microsoft.com/en-us/library/cc678863(v=office.14).aspx#Section3

http://blogs.msdn.com/b/russmax/archive/2010/01/08/changing-sharepoint-2010-service-accounts.aspx

Thanks,

Russ Maxwell, MSFT

How to properly change the Host Header URL of a web application in SharePoint 2010

2013-02-28T02:08:21Z

I’m baaaaackkkkkkk, lol. I apologize for taking about a 6 month break but I’m back and will have some new fresh blog content starting now. I recently ran into an interesting scenario with a request to change the Host Header URL of a web application in SharePoint 2010. I was surprised that I didn’t find much out on TechNet or any Microsoft publications on how to accomplish this properly for SharePoint 2010 which is the purpose of this blog. First, let’s describe what it means to create a host header web application. When you create a web application you have the ability to add a host header url.

In the above screenshot, I created a web application with a host header URL of RussMaxOriginalHost. After creating the web application and site collection, I browse to the IIS Site and look at the bindings and the Host Header is added automatically which is expected.

Also, If I check Alternate Access Mappings in Central Admin I see the default zone is populated with my host header URL for the given web application

And finally, browsing to the site after creating the site collection renders the site with the specified host header.

What happens when I need to change this host header URL value? My original thought was to simply go edit the bindings of IIS and change the alternate access mapping. This actually works but isn’t recommended because eventually the bindings in IIS will be overwritten by the original host header URL.

For Example:

Going to Central Administrator, Services on Server, and stopping and starting the Microsoft SharePoint Foundation Web Application service for the specified server will overwrite any manually updated host header entry in IIS/Site bindings with the original host header URL.

Adding a new SharePoint Server to an existing farm will get the site with the original host header URL after Microsoft SharePoint Foundation Web Application service is started which happens automatically via PSConfig.

Explanation

When you create a web application and specify a host header URL, you stamp the host header URL on the spwebapplication object which is stored in the objects table in the configuration database. When you try to change the host header URL by updating the Alternate Access Mapping and manually update the bindings in IIS, the SPWebApplication object is never updated so it continues to maintain knowledge of the original host header URL and not the new one. When you start the Microsoft SharePoint Foundation Web Application service in Central Admin, the information from the associated SPWebApplication object in the configuration database is used when instantiating IIS Sites. This is why the original host header URL shows up.

Options

The best automated way to update the Host Header URL on a web application is similar to the approach in SharePoint 2007 which is to un-extend and re-extend the Web Application. That is, disassociate any connection with the existing IIS site and re-establishing that association. Steps are the following assuming I’m changing my host header URL from russmaxoriginalhost to russmaxnewhost.

1. Go to Central Administrator, Application Management, Select the Web Application you wish to change and hit the arrow under Delete and choose Remove SharePoint from IIS Site.

2. Next, select the associated site you which to update the host header URL against. For Delete IIS web sites, select No and hit OK.

Note: If you have more than two IIS sites available, that means your web application is extended so choose the appropriate Zone/IIS Site that you would like to update.

3. Go to Central Administrator, Application Management, Select the same Web Application and choose Extend button on the ribbon.

4. Choose Use an exisitng IIS Site and choose the original site from Step 2 above. Specify the New Host Header URL and any other settings that should match. In my case, I updated the Host Header to RussMaxNewHost and hit OK here.

After updating DNS, the site comes up with no problems.

Question: What if the web application is already extended to another zone let’s say extranet zone and you want to change host header URL for the default zone?

Answer: This will still work fine since your only making changes to the IIS site associated with the default zone, the site hosting the extranet zone will remain untouched and should continue to work fine after the above changes.

Question: How can I validate the associated SPWebApplication object in the configuration database contains the new host header URL?

Answer: You can run the following Power Shell to validate. Here is a sample:

$wa = get-spwebapplication “http://randomURL”
$iissettings = $wa.GetIisSettingsWithFallback("Default")
$iissettings.serverbindings

This will output the host header URL for the default zone for the web application of your desired URL.

Thanks,

Russ Maxwell, MSFT

Want to call SharePoint 2010 web services within browser based InfoPath 2010 forms?

2012-08-17T14:20:00Z

I recently ran across an interesting issue where InfoPath forms were calling OOB SharePoint 2010 web services. In particular, these forms contained one or more fields which called SharePoint OOB web service. These forms were published to web applications configured with Claims Authentication. Opening this form works fine as long as you open the form using InfoPath on the client. When attempting to view the form in the browser the following error occurs:

Subsequently, you will see an error like the following in the ULS logs:

The following query failed: GetUserProfileByName (User: 0e.t|exo-sts|user1@consoto.com, Form Name: otherForm, IP: , Connection Target: , Request: http://wfe1.contoso.com/_layouts/FormServer.aspx?XsnLocation=https://wfe1.contoso.com/sites/siteA/FormServerTemplates/IT WAR.xsn&SaveLocation=https://wfe1.contoso.com/sites/siteA/ITWARdir&ClientInstalled=t=1, Form ID: urn:schemas-microsoft-com:office:infopath:FormLib:-myXSD-2012-07-12T16-05-43 Type: DataAdapterException, Exception Message: The remote server returned an error: (500) Internal Server Error. Server was unable to process request. ---> Attempted to perform an unauthorized operation. The remote server returned an error: (500) Internal Server Error.)

The quick answer is this is a limitation of InfoPath Form Services not being claims aware.

Basics of Building InfoPath Forms that call Web Services

Before discussing any workarounds, I’d like to provide a brief overview of calling SharePoint web services within an InfoPath form. First, you always start off with an InfoPath form template which is an .xsn extension and you add labels and controls.

For Example:

On the left we have labels to provide a description of each field. On the right we can insert controls like a text box for example.

Note: The far right fields represent the controls I’ve inserted on the left which is why I have 3.

Assuming you want to call the UserProfile web service to automatically retrieve and populate the control next to User Name, define a Data Connection within InfoPath to this web service.

Question: So how do you setup a Data Connection to call an OOB SharePoint web service?
Answer: An excellent blog walks you through this process. Click Here!

After reviewing the blog above, I assume you know how to build a basic data connection to point to a SharePoint web service. Now clicking on Manage Data Connections in my InfoPath form:

Also, field1 is bound to the data connection pictured above:

Publish the form template to a SharePoint Form Library

The final step is saving the template and publishing it to a Form Library on the desired Site Collection.

1. Select File, Publish, Publish Form to a SharePoint Library
2. Follow the publishing wizard and enter URL of the Site Collection and click next
3. Check “Enable this form to be filled out by using a browser” and select Form Library, click next
4. Create a Form Library or choose from existing and click next
5. Click next and then Publish

This basic method of publishing InfoPath forms works fine with Data Connections but it doesn’t work well when publishing the form to Claims based Web Applications if the data connections are used to call SharePoint web services.

Work Around

Update the web service URL in the data connection to call a Web Application configured to use Classic Authentication.

The form must make calls to a different web application configured with Classic Windows Authentication.

For Example:

Both Web Applications reside in the same farm.
Web Application A (Claims Based Authentication)
Web Application B (Classic Windows Authentication)
InfoPath Form is published to Web Application A
The published form calls OOB SharePoint web services on Web Application B

If the form must make web service calls across Web Applications, the following must occur:

(This step is necessary but not listed below) Provision a new web application with Windows Classic Authentication

1. Update the data connection web service URL to call the Web App URL’s configured with Classic Auth
2. Create a data connection library in a given site collection
3. The data connection must be converted to file of type UDCX and published to a Data Connection Library
4. Download the UDCX from the data connection library and edit it
5. The form must be full trusted and published locally
6. The form template and data connection file are uploaded to SharePoint Central Administrator
7. Adjust configuration settings within Central Admin InfoPath Forms Services
8. The form template is then activated to a Claims Web App\Site Collection
9. Test to ensure it works without error

Detailed Steps are below

Note: I assume you know how to provision a new web application with classic authentication so skipping this step but make sure you have one provisioned.

1. Update the data connection web service URL to call the new Web App URL configured with Classic Authentication

a. Open the form template (.xsn) with InfoPath Designer 2010

b. Select Manage Data Connections on bottom right under actions and select Modify on the data connection and update the web service endpoint to the Classic Web App URL

For Example:

c. After clicking next, select the web service method which in this case I chose GetUserProfileByName and click Next.
d. Click Next through the next couple of prompts until you get to here:

e. Ensure you leave Automatically retrieve data when form is opened checked and click Finish

2. Create a data connection library in a given site collection

a. This is for temporary usage and can be deleted at the end of these steps
b. For a site collection select Site Actions\More Options Filter by Library and Choose Data Connection Library
c. Name it and click Create

3. The data connection must be converted to file of type UDCX

a. After clicking Finish on step 1, select Convert to Connection File

b. Enter the URL + filename of the Data Connection Library you created from step 2 and hit OK

Now the udcx file is converted and stored in your data connection library:

4. Download the UDCX from the data connection library and edit with notepad

a. Download the UDCX file from the Data Connection Library you uploaded
b. I put mine in c:\IPStuff
c. Edit the UDCX File locally and add\save the following:

<udc:Authentication>
<udc:UseExplicit CredentialType="NTLM">
 <udc:UserId>domain\user</udc:UserId>
 <udc:Password>password</udc:Password>
 </udc:UseExplicit>
</udc:Authentication>

Note: Ensure you use an account name and password with appropriate permissions to the remote web app

5. The form must be full trusted and published locally

a. Within the Form, select File, Info and Choose Form options
b. Ensure you uncheck “Automatically determine security level and choose Full trust and Click OK

c. Select File, Publish, and choose SharePoint Server
d. Type a URL for a form library (we won’t actually publish it to the library), click next
e. Ensure you leave “Enable this form to be filled out by using a browser” checked and click next

(See the following screenshot)

f. Next specify a location on the local drive with a filename.xsn extension

g. Hit next through the next few prompts and then click finish.

6. The form template and data connection file are uploaded to SharePoint Central Administrator

a. Upload the form template saved from step 4 to Central Administrator\General Application Settings\Upload Form Template.
b. Browse to form and click Upload
c. Upload the data connection (UDCX file) to Central Administrator\General Application Settings\Manage Data Connection Files
d. Select the file and ensure Allow HTTP Access to this file is checked and click upload

7. Adjust configuration settings within Central Admin InfoPath Forms Services

a. Access Central Administrator\General Application Settings\Configure InfoPath Forms Services
b. Check the following options:

Check:   Allow users to browser-enable form templates
Check:   Render form templates that are browser-enabled by users
Check:   Require SSL for http authentication to data sources
Check:   Allow User form templates to use authentication information contained in data connection files
Check:   Allow cross-domain data access for user form templates that use connection settings in a data connection file

It should look like:

c. Finally hit OK

8. Activate the form to a Site Collection hosted by a Claims Based Web Application

a. Access Central Administrator\General Application Settings\Manage Form Templates
b. Select the new form template and Activate to a Site Collection

c. Choose the appropriate site collection and hit OK

9. Test to ensure the form works without error

a. Test by going to the targeted site collection’s formservertemplates:

For Example: https://siteA/formservertemplates

b. Choose edit in browser for the newly published form.

In the screenshot above, my form was published to claims based web application at site collection:

https://smokewfe

The form opened in the browser and the User Name field was correctly populated the FirstName attribute after calling the UserProfile web service that is hosted by the Web Application configured with Classic Authentication

Note: It might fail the first couple of times but after 2 minutes or so it works fine which I assume is a result of the application pool spinning up.

Q/A section:

Question: Why can’t I just update the Form’s Data Connection to point the web service URL endpoint to the Classic Windows Auth Web Application and republish it back to the original location which is a Site Collection\Form library on the Claims Based Web Application?

Answer: In order to call a web service on a remote Web Application the InfoPath form must be fully trusted. Forms that are fully trusted require administrator approval and thus must be published to Central Administrator.

Question: Why do I need to convert the data connection to file?
Answer: Because in order to make “cross-domain” connections you must use a data connection file.

Question: Do I still need the Data Connection Library created from Step 2 above?
Answer: No, it was a temporary holding place so now that the data connection file is stored on Central Admin, you can remove it from the site collection if it’s no longer needed.

Resources:

http://blogs.msdn.com/b/infopath/archive/2006/10/02/data-connections-in-browser-forms.aspx
http://blogs.msdn.com/b/bowerm/archive/2006/12/06/calling-web-services-from-infopath-web-forms.aspx
http://blogs.microsoft.co.il/blogs/itaysk/archive/2007/04/05/InfoPath-_2D00_-Get-the-current-user-without-writing-code.aspx
http://technet.microsoft.com/en-us/library/cc262921.aspx

Thanks,

Russ Maxwell, MSFT

SharePoint PowerShell Script Series Part 7: Updating or Viewing the Quick Launch Bar

2012-06-13T02:42:00Z

SharePoint Administrator’s running this script want to perform the following tasks for a specified site:

1. Add new Headings and Links to the Quick Launch Bar
2. Add new links to existing Headings on the Quick Launch Bar
3. View the Quick Launch layout for a given Site

Question: What is the quick launch bar?

Answer: The quick launch bar resides on the left hand side of a SharePoint site and consists of Headings and Links. It’s an easy way for users to quickly navigate to internal SharePoint data like other sites, lists, or document libraries. It’s also an easy way for users to navigate to external SharePoint content like an external web site.

Quick Launch bar is circled in red:

It’s possible to update the quick launch bar by going to Site Actions\Site Settings and under Look and Feel section, click on Navigation link. If you don’t have a navigation link, you’ll need to enable the SharePoint Server Publishing Infrastructure feature which is also under Site Actions\Sites Settings and under Site Collection Administration, click on Site Collection Features. After clicking on Navigation link, you’re taken to the areanavigationsettings.aspx page. Within this page, you have a nice navigation control to Add, Delete, or Move Headings and/or Links.

Note: Everything under the Current Navigation reflects the Quick Launch bar layout.

Question: Do sub-sites automatically inherit the current navigation “quick launch bar” from the root site?
Answer: No, by default when creating a sub-site using team site template, navigation isn’t inherited from the root.

Question: Can I control what whether or not a subsite inherits the quick launch bar from the parent site?
Answer: Yes, in fact from the navigation page I mentioned above is the following section:

Note: The picture above has the default setting selected which is not to inherit from the parent.

Intro into the Quick Launch PowerShell script

First, I really don’t know why I wrote this script other than being curious on how to do it. I wanted to use PowerShell to add Headings and Links to the quick launch bar in an automated fashion. I also wanted the ability to view the quick launch bar.

Before running this PowerShell Script, a couple of important things to point out:

1. If you add headings and/or links to a parent site, any subsite inheriting current navigation from the parent will also see the newly added headings and/or links.

2. If you choose to add headings and/or links to a subsite, the script will only add to subsites that have current navigation set to the following:

Display only the navigation items below the current site

Scenario 1: Add new heading and links to parent site

In this scenario, I’m adding a new heading with 2 links in the parent site or root web in the Site Collection. The heading will be external and will contain two external links to MSDN and Technet.

Setup looks like:

External

Heading Name: MSFT Resources

Link Name: MSDN
Link URL: http://msdn.microsoft.com

Link Name: TechNet
Link URL: http://technet.microsoft.com

1. Prompt to run against root site or subsite and I choose the root:

2. Prompted to enter the URL and hit enter

3. Prompted to choose from 3 options. I chose option 1.

4. Prompted to choose external or internal links. I chose External

5. Prompted to type in name of new heading and press enter.

6. Prompted to type in # of links to add.

7. Prompted to enter name and URL for each link.

8. Get response Script is completed:

9. Final result:

Scenario 2: Add new two links to a sub site

In this scenario, I’m adding 2 internal links to an existing heading called Departments that resides in a subsite. The heading is internal and I’ll be adding to internal links that point to two lists that reside in the subsite.

Setup looks like:

Internal

Existing Heading Name: Departments

Link Name: Sales
Link URL: /subsite/lists/sales/allitems.aspx

Link Name: IT
Link URL: /subsite/lists/IT/allitems.aspx

1. Prompt to run against root site or subsite and I choose the subsite.

2. Prompted to enter the URL and hit enter

3. Next, I choose option 2 to add new links to existing heading.

4. Next, I choose 1 for Internal links

5. Next, I choose Option 2 to create links directly under a heading:

6. Prompted to choose existing heading to create links under. I chose 4:

7. Now I choose the # of links, link names, and internal relative URL to both lists:

8. Script is complete!

9. Final Result:

Scenario 3: Viewing Quick Launch bar of parent site

In this scenario, I’m viewing the Quick Launch bar of parent site by running the script. Currently, the parent site displays the following quick launch bar:

Most of the steps are identical except I choose option 3 to view quick launch bar:

The final output provides the Header and associated links:

Note: This PowerShell script is tested only on SharePoint 2010

Instructions for running the script:

1. Copy the below script and save it in notepad
2. Save it with a anyfilename.ps1 extension
3. To run, copy the file to a SharePoint Server
4. Select Start\Microsoft SharePoint 2010 Products\SharePoint 2010 Management Shell
5. Browse to directory holding the copied script file
6. Run the script: .\anyfilename.ps1 (assuming anyfilename is the name of the file)

<# ==============================================================
//
// Microsoft provides programming examples for illustration only,
// without warranty either expressed or implied, including, but not
// limited to, the implied warranties of merchantability and/or
// fitness for a particular purpose.
//
// This sample assumes that you are familiar with the programming
// language being demonstrated and the tools used to create and debug
// procedures. Microsoft support professionals can help explain the
// functionality of a particular procedure, but they will not modify
// these examples to provide added functionality or construct
// procedures to meet your specific needs. If you have limited
// programming experience, you may want to contact a Microsoft
// Certified Partner or the Microsoft fee-based consulting line at
// (800) 936-5200.
//
// For more information about Microsoft Certified Partners, please
// visit the following Microsoft Web site:
// https://partner.microsoft.com/global/30000104
//
// Author: Russ Maxwell (russmax@microsoft.com)
//
// ---------------------------------------------------------- #>

[Void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint")
Start-SPAssignment -Global

function question1()
{
    Write-Host "Option 1: Quick Launch: Add new heading with link\links"
    Write-Host "Option 2: Quick Launch: Add new link\links to existing heading"
    Write-Host "Option 3: Quick Launch: View all Quick Launch Bar headings/links for specified Site"
    $res2 = Read-Host "Press 1,2 or 3 and hit Enter"
    return $res2
}

function question2()
{
    ##More Questions##
    Write-Host "Will Links point to internal SharePoint data or External data outside of SharePoint?"
    Write-Host "Option 1: Internal"
    Write-Host "Option 2: External"
    $res3 = Read-Host "Select 1 or 2 and press Enter"
    return $res3
}

function addQuickLaunch()
{
    ##If True, Adding Heading and Links##
    if($res2 -eq 1)
    {
        $empstr = ""
        if($res3 -eq 1)
        {
            $navheadnode = New-Object Microsoft.SharePoint.Navigation.SPNavigationNode($headstr, $empstr, $false)
            Write-Host
            Write-Host
        }

        elseif($res3 -eq 2)
        {
            $navheadnode = New-Object Microsoft.SharePoint.Navigation.SPNavigationNode($headstr, $empstr, $true)
        }

        $navquicklaunch.addaslast($navheadnode)
        $navheadnode.properties["BlankUrl"] = $true
        $navheadnode.properties["NodeType"] = "Heading"
        $navheadnode.update()
        Write-Host
        Write-Host
        [Microsoft.SharePoint.Navigation.SPNavigationNodeCollection] $childcoll = $navheadnode.children
        addLinks
    }

    ##If True, Adding links under an existing heading##
    elseif($res2 -eq 2)
    {
        Write-Host "Option 1: Create links directly under the Quick Launch bar"
        Write-Host "Option 2: Create links directly under a heading"
        $res4 = Read-Host "Press option 1 or 2 and hit Enter"

        if($res4 -eq 1)
        {
           [Microsoft.SharePoint.Navigation.SPNavigationNodeCollection] $childcoll = $navquicklaunch.children
           addlinks
        }

        elseif($res4 -eq 2)
        {
            Write-Host
            Write-Host "Choose the heading to place links under"
            [Microsoft.SharePoint.Navigation.SPNavigation] $nav2 = $web.Navigation
            [Microsoft.SharePoint.Navigation.SPNavigationNodeCollection] $quickLaunch2 = $nav2.quicklaunch
            $ctr1 = 1

            foreach($node in $quickLaunch2)
            {
                Write-Host "Option "$ctr1 ". " $node.Title -ForegroundColor Magenta
                 $ctr1++
            }
        }

        $res5 = Read-Host "Select the appropriate option and press enter"
        if($res5 -ne $null)
        {
            $ctr1 = 1
            foreach($node in $quickLaunch2)
            {
                if($res5 -eq $ctr1)
                {
                  [Microsoft.SharePoint.Navigation.SPNavigationNodeCollection] $childcoll = $node.Children
                 addlinks
                }
                $ctr1++
            }
        }
    }
}

###########################
##Function adds the links##
###########################
function addLinks
{
    ##Create arrays of names\links##
    $linknamearr = @()
    $linkurlarr = @()

    Write-Host "Please type the # of links you would like to add"
    $linknum = Read-Host
    $ctr = 1

    while($ctr -le $linknum)
    {
        $navlinkname = Read-Host "Enter Name for link: " $ctr "(Press Enter)"
        $linknamearr += ,$navlinkname

        if($res3 -eq 1)
        {
           $navlinkurl = Read-Host "Enter Internal relative URL to SharePoint Data and press enter"
           $linkurlarr += ,$navlinkurl
         }
         elseif($res3 -eq 2)
         {
            $navlinkurl = Read-Host "Enter External URL for link and press Enter"
            $linkurlarr += ,$navlinkurl
         }
        Write-Host
        Write-Host
        $ctr++
    }

    ###Now iterate through arrays and add nodes###
    $ctr = 0

    while($ctr -lt $linknum)
    {
      if($res3 -eq 1)
         {
          $navlinknode = New-Object Microsoft.SharePoint.Navigation.SPNavigationNode($linknamearr[$ctr], $linkurlarr[$ctr], $false)
         }

      elseif($res3 -eq 2)
        {
          $navlinknode = New-Object Microsoft.SharePoint.Navigation.SPNavigationNode($linknamearr[$ctr], $linkurlarr[$ctr], $true)
        }

        $childcoll.addaslast($navlinknode)
        $navlinknode.isVisible = $true
        $navlinknode.Properties["vti_navsequencechild"] = $false
        $navlinknode.Properties["NodeType"] = "AuthoredLinkPlain"
        $navlinknode.update()
         $ctr++
    }
}

####################################
##Function to display quick launch##
####################################
function displayQuickLaunch()
{
    [Microsoft.SharePoint.Navigation.SPNavigation] $nav = $web.Navigation
    [Microsoft.SharePoint.Navigation.SPNavigationNodeCollection] $quickLaunch = $nav.quicklaunch

    foreach($node in $quickLaunch)
    {
        Write-Host
        Write-Host
        Write-Host "Header: " $node.Title -ForegroundColor Cyan
        $subnodeColl = $node.Children

        foreach($subnode in $subnodeColl)
        {
          Write-Host "Link Name: " $subnode.title -foregroundcolor Magenta
          Write-Host "Link URL: " $subnode.url -foregroundcolor Magenta
          Write-Host
        }
    }
}

######################################
##Quick Launch Script Starts Here#####
######################################
Write-Host "Option 1: Modify or View QuickLaunch for root site in a given Site Collection"
Write-Host "Note: Making changes to the root of the Site Collection will push those changes to all subsites inheriting Navigation." -ForegroundColor Red
Write-Host
Write-Host "Option 2: Modify or View QuickLaunch for a subsite that isn't inheriting navigation from the root site"
Write-Host
$res1 = Read-Host "Press 1 or 2 and hit Enter"
Write-Host

if($res1 -eq 1)
{
    Write-Host "Enter the URL of the Desired Site Collection and press Enter"
    $siteURL = Read-Host
    $site = Get-SPSite $siteURL
    $web = $site.rootweb

    if($web -ne $null)
    {
        $navobj = $web.Navigation
        $navquicklaunch = $navobj.QuickLaunch
        $res2 = question1
        Write-Host

        if($res2 -eq 1)
        {
            $res3 = question2
            Write-Host
            Write-Host "Please type the name of the new heading and press Enter"
            $headstr = Read-Host
            Write-Host
            ##Calling function which will add the heading/links##
            addquickLaunch
            Write-Host
            Write-Host "Script is completed" -ForegroundColor Green
        }

        elseif($res2 -eq 2)
        {
            $res3 = question2
            Write-Host
            ##Calling function which will add the heading/links##
            addQuickLaunch
            Write-Host
            Write-Host "Script is completed" -ForegroundColor green

        }

        elseif($res2 -eq 3)
        {
          displayQuickLaunch
        }
    }
    else
    {
        Write-Host
        Write-Host
        Write-Host "The Site is null or you entered an invalid URL" -ForegroundColor Red
        Write-Host "Script completed making 0 changes" -ForegroundColor Green
    }
}

elseif($res1 -eq 2)
{
    Write-Host "Enter the url of the Subsite and hit Enter"
    $subUrl = Read-Host
    $web = Get-SPWeb $subUrl
    $site = $web.site
    if(($web -ne $null) -and ($web.url -ne $site.rootweb.url))
    {
        $res2 = question1
        Write-Host
        ###Ensure navigation isn't inheriting from parent or isn't set to show siblings###
        if(($web.allproperties["__InheritCurrentNavigation"] -eq "false") -and ($web.allproperties["__NavigationShowSiblings"] -eq "false"))
        {
            $navobj = $web.Navigation
            $navquicklaunch = $navobj.QuickLaunch

            if($res2 -eq 1)
            {
              $res3 = question2
              Write-Host
              Write-Host "Please type the name of the new heading and press Enter"
              $headstr = Read-Host
                addQuickLaunch
              Write-Host
              Write-Host

              Write-Host "Script Complete!" -ForegroundColor Green
            }
            elseif($res2 -eq 2)
            {
              $res3 = question2
              Write-Host
              ##Calling function which will add the heading/links##
              addQuickLaunch
              Write-Host
              Write-Host "Script is completed" -ForegroundColor green
            }

            elseif($res2 -eq 3)
            {
              displayQuickLaunch
            }
        }


        ##Checking to see if inheriting navigation##
        elseif($web.allproperties["__InheritCurrentNavigation"] -eq "true")
        {
            Write-Host "Navigation is inheriting from parent web!" -ForegroundColor Red
            Write-Host "Script completed making 0 changes" -ForegroundColor Green
        }

        ##Check to see if navigation is set to share siblings##
        elseif($web.allProperties["__NavigationShowSiblings"] -eq "true")
        {
            Write-Host "Navigation isn't inheriting but is displaying site siblings" -ForegroundColor Red
            Write-Host "Script completed making 0 changes" -ForegroundColor Green
        }

    }
    else
    {
        Write-Host
        Write-Host
        Write-Host "The Site is null or is the Parent Web" -ForegroundColor Red
        Write-Host "Script completed making 0 changes" -ForegroundColor Green
    }
}

Stop-SPAssignment -Global

Thanks!

Russ Maxwell, MSFT

SharePoint PowerShell Script Series Part 6 - Is Anonymous Access Enabled?

2012-04-21T22:54:08Z

SharePoint Administrator's that run this script usually want to answer one of the following questions:

    1. Is Anonymous Access enabled anywhere in my Site Collection. If so, where?
    2. I want to confirm Anonymous Access is enabled in a specific location
    3. I want to collect a record of how Anonymous Access is setup for an entire Site Collection.

Anonymous Access can be set at 3 different levels.

Web Application
This is configurable in Central Administrator:

Site
This is configurable within a given Site by selecting Site Actions, Site Permissions, and click the Anonymous Access button from the Ribbon:

List\Document Library
Assuming you configured Lists and Libraries option at the Site Level (above screenshot), then you can enable Anonymous Access for any given List or Document Library belonging to that site. To do this: access the specified list and select the following from the ribbon:

Library\Library Permissions Button\Anonymous Access Button

Note: Anonymous Access cannot be set on a specific item in a list.

Checking For Anonymous Access (Out of the Box)
At the very least, you want to check if the Web Application and Specified sites are enabled with Anonymous Access.
If you check the site level and Anonymous is set for "Lists and Libraries", then you would further inspect the desired List or Library to see if Anonymous Access is enabled (above screenshot).

If you've been through this exercise before, your well aware this can be a cumbersome and time consuming task if you have hundreds of sites and hundreds or even thousands of lists in each site for a given Site Collection.

Checking for Anonymous Access (Automatically)
I wrote a PowerShell script to do the heavy lifting and traverse all Sites, Lists, and Libraries in a given Site Collection to check if Anonymous Access is enabled at any level. If Anonymous Access is enabled at any level, a log file in CSV format with more information on what's enabled and where it's enabled.

How does the script work? Check the following scenario:

Scenario: Single Site Collection contains four sites.

Site 1: http://wfe:3131
Site Level: Anonymous Access is set for Lists and Libraries
List\Library: I configured one list and one document library with Anonymous Access

Site 2: http://wfe:3131/SubSite
Site Level: Anonymous Access is set as "Entire Web Site"

Note: The other 2 sites were created off of Site 1 since it's a publishing site and inherits the settings from the root site (Site 1 - above).

Running the PowerShell Script looks like: (Yes, I used Magenta)

From the screenshot above, notice Anonymous Access is Enabled so a CSV file was created which looks like:

Let’s discuss the columns from the above screenshot in more detail:

Column - Level: The level indicates at what level Anonymous Access is enabled. If Level equals "Site Level: Lists and Libraries", then we will go and check the List and Libraries to see if Anonymous Access is enabled. As you can see, Anonymous Access is Enabled and configured on two lists.

Column - URL: The URL where Anonymous is enabled. If the level is Web Application, it will only put the name of the Web Application where it's enabled in the URL field.

Configured List\Lib: This column will flag yes if Anonymous Access is enabled and configured for a given List\Library. What I mean by configured is the view items check box is checked for Anonymous Access. (See screenshot under List\Document Level above.)

Note: This PowerShell script is tested only on SharePoint 2010

Instructions for running the script:

<# ==============================================================
//
// Microsoft provides programming examples for illustration only,
// without warranty either expressed or implied, including, but not
// limited to, the implied warranties of merchantability and/or
// fitness for a particular purpose.
//
// This sample assumes that you are familiar with the programming
// language being demonstrated and the tools used to create and debug
// procedures. Microsoft support professionals can help explain the
// functionality of a particular procedure, but they will not modify
// these examples to provide added functionality or construct
// procedures to meet your specific needs. If you have limited
// programming experience, you may want to contact a Microsoft
// Certified Partner or the Microsoft fee-based consulting line at
// (800) 936-5200.
//
// For more information about Microsoft Certified Partners, please
// visit the following Microsoft Web site:
// https://partner.microsoft.com/global/30000104
//
// Author: Russ Maxwell (russmax@microsoft.com)
//
// ---------------------------------------------------------- #>

[Void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint")

Start-SPAssignment -Global

######################################
##Creating and Returning a DataTable##
######################################
function createDT()
{
    ###Creating a new DataTable###
    $tempTable = New-Object System.Data.DataTable

    ##Creating Columns for DataTable##
    $col1 = New-Object System.Data.DataColumn("Anonymous Access")
    $col2 = New-Object System.Data.DataColumn("Level")
    $col3 = New-Object System.Data.DataColumn("URL")
    $col4 = New-Object System.Data.DataColumn("Configured List\Lib")

    ###Adding Columns for DataTable###
    $tempTable.columns.Add($col1)
    $tempTable.columns.Add($col2)
    $tempTable.columns.Add($col3)
    $tempTable.columns.Add($col4)

    return ,$tempTable
}

#####################################
##Check WebApp for Anonymous Access##
#####################################
function checkwebappAnon()
{
    $webAnon = $site.IISAllowsAnonymous.tostring()
    $tempanonCheck = 0;
    if ($webAnon -eq "true")
    {
        #Add a row to DataTable
        $row = $dTable.NewRow()
        $row["Anonymous Access"] = "Enabled"
        $row["Level"] = "WebApplication"
        $row["URL"] = $site.WebApplication.Name
        $dTable.rows.Add($row)
    }

}

######################################
##Check the Site for Anonymous Access#
######################################
function checksiteAnon()
{
    $tempanonCheck = 0
    $checkWeb = $web.AllowAnonymousAccess.tostring()
    $checkWebState = $web.AnonymousState.tostring()
    $webMask = $web.AnonymousPermMask64.tostring()
    Write-Host
    Write-Host "Checking how Anonymous is set up on site: " $web.Url -ForegroundColor Magenta

    if(($checkWeb -eq "True") -and ($checkWebState -eq "On"))
    {
        #Add a row to DataTable#
        $row = $dTable.NewRow()
        $row["Anonymous Access"] = "Enabled"
        $row["Level"] = "Site Level: Entire WebSite"
        $row["URL"] = $web.Url.tostring()
        $dTable.rows.Add($row)
        $tempResult = 1
    }

    elseif(($checkWeb -eq "False") -and ($checkWebState -eq "Enabled") -and ($webMask -eq "Open"))
    {
        #Add a row to DataTable#
        $row = $dTable.NewRow()
        $row["Anonymous Access"] = "Enabled"
        $row["Level"] = "Site Level: Lists and Libraries"
        $row["URL"] = $web.Url.tostring()
        $dTable.rows.Add($row)
        $tempResult = 2
    }

    else
    {
        $tempResult = 3
    }

    return $tempResult
}

############################################
##Check List\Libraries for Anonymous Access#
############################################
function checklistAnon()
{
###Checking each list and library for anonymous access###
$lists = $web.lists
$count1 = $lists.count
$hasAnon = 0

Write-Host "Checking " $lists.count " lists\libaries for Anonymous Access" -ForegroundColor Magenta

###Setting String Vars###
$defMask1 = "OpenWeb"
$defMask2 = "EmptyMask"
$defTax = "TaxonomyHiddenList"

foreach($list in $lists)
{
    $listUrl = $web.url + "/" + $list.Title
      $listMask = $list.AnonymousPermMask.tostring()
    $tax = $list.Title.ToString()

    ##Checking List eventhough Anonymous Access was disabled at SPWeb Level##
    if(($webResult -eq '3') -and ($defTax.CompareTo($tax) -ne '0'))
    {
          if($listMask.CompareTo($defMask2) -ne '0')
        {
          if($listMask.CompareTo($defMask1) -eq '0')
             {
            #Anonymous Access is Enabled but not Configured on list\library#
               $row = $dTable.NewRow()
            $row["Anonymous Access"] = "Enabled"
            $row["Level"] = "List\Library"
            $row["URL"] = $listUrl
            $row["Configured List\Lib"] = "No"
            $dTable.rows.Add($row)
            $hasAnon++
           }
          else
             {
           #Anonymous Access Enabled and Configured on list\library#
               $row = $dTable.NewRow()
            $row["Anonymous Access"] = "Enabled"
            $row["Level"] = "List\Library"
            $row["URL"] = $listUrl
            $row["Configured List\Lib"] = "Yes"
            $dTable.rows.Add($row)
               $hasAnon++
          }
        }
    }

     elseif(($webResult -eq '2') -and ($defTax.CompareTo($tax) -ne '0'))
        {
          if(($listMask.CompareTo($defMask2) -ne '0') -and ($listMask.CompareTo($defMask1) -ne '0'))
        {
           #Anonymous Access Enabled and Configured on list\library#
               $row = $dTable.NewRow()
            $row["Anonymous Access"] = "Enabled"
            $row["Level"] = "List\Library"
            $row["URL"] = $listURL
            $row["Configured List\Lib"] = "Yes"
            $dTable.rows.Add($row)
            $hasAnon++
        }
      }
    $count1--
    if($count1 % '10' -eq '0')
    {
     Write-Host "Total # of lists\libraries left to check: " $count1 -ForegroundColor DarkYellow
    }
}
Write-Host
Write-Host "Total # of lists\libraries with Anonymous Access Enabled: " $hasAnon -ForegroundColor Cyan
}

########################
###Script Starts Here###
########################
$output = Read-Host "Enter a location for the output file (For Example: c:\logs\)"
$filename = Read-Host "Enter a filename"
$url = Read-Host "Please enter the URL of desired site collection and press enter"

###Getting a new DataTable###
[System.Data.DataTable]$dTable = createDT

###Getting Site Collection###
$site = Get-SPSite $url

###Checking if WebApp has Anonymous set###
checkwebappAnon

###Gathering web collection###
$webs = $site.Allwebs
$count = $webs.Count
Write-Host "Checking for Anonymous Access on " $count " Sites" -ForegroundColor Magenta

foreach($web in $webs)
{
    $webResult = 0
    ###calling function to check anonymons on spweb###
    $webResult = checksiteAnon

    if(($webResult -eq '2') -or ($webResult -eq '3'))
    {
        Write-Host "Checking for Anonymous Access on List and Libraries" -ForegroundColor Magenta
        ###calling function to check anonymons on lists and libs###
        checklistAnon
    }

    $count--

    if($count -ne '0')
    {
      Write-Host
      Write-Host "Total # of sites left to check: " $count -ForegroundColor DarkYellow
    }

    else{Write-Host "Operation Completed" -ForegroundColor DarkYellow}
}

if($dTable -ne $null)
{
    $name = $output + "\" + $filename + ".csv"
    $dTable | Export-Csv $name -NoTypeInformation
    Write-Host "Anonymous Access was detected" -ForegroundColor Green
    Write-Host "Log File Created: " $name
}
else
{
    Write-Host "Anonymous Access is Disabled for the entire Site Collection" -ForegroundColor Green
            Write-Host "No Log File Created" -ForegroundColor Green
}

Write-Host "Script Complete"

Stop-SPAssignment -Global

Thanks!

Russ Maxwell, MSFT

Behind the scenes: Opening a Document from a SharePoint 2010 Document Library

2012-03-10T18:51:18Z

This is a summary to provide a glimpse of the Client to Server interaction when a user attempts to open a word document from a SharePoint 2010 Document Library. A lot of decision making goes behind each attempt when factoring in client Operating System level and Office version. In this blog, we will cover what things look like with the following setup:

Client: Windows 7 using Office 2010 “Word 2010”
Server: SharePoint 2010
Site Name: http://wfe:9999
Document LIbrary Name: Shared Documents
Filename: Test8.docx

I captured a network trace using Network Monitor and will output the frame details to describe the interaction. I recommend this as a first step if your attempting to troubleshoot problems when opening Word documents.

1. User first Authenticates into the Site via HTTP/HTTPS and attempts to open a document from a document library

2. Once Authenticated, the protocol client issues an HTTP OPTIONS request to the server again to determine the capabilities of the server.

A network trace of the client request looks like:

      Frame: Number = 155, Captured Frame Length = 841, MediaType = ETHERNET
+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[00-15-5D-21-17-0A],SourceAddress:[02-00-4C-4F-4F-50]
+ Ipv4: Src = 10.1.0.5, Dest = 10.1.0.12, Next Protocol = TCP, Packet ID = 14378, Total IP Length = 827
+ Tcp: Flags=...AP..., SrcPort=46989, DstPort=9999, PayloadLen=787, Seq=892234633 - 892235420, Ack=3231455964, Win=32850 (scale factor 0x2) = 131400
- Http: Request, OPTIONS /Shared%20Documents/ , Using NTLM Authorization
    Command: OPTIONS
+ URI: /Shared%20Documents/
    ProtocolVersion: HTTP/1.1
    UserAgent: Microsoft Office Protocol Discovery
    Host: wfe:9999
+ Authorization: NTLM
    Connection: Keep-Alive
    ContentLength: 0
    HeaderEnd: CRLF

3. The server responds to the HTTP Options request and provides a list of supported methods and also provides values in the X-MSFSSHTTP header and the MS-Author-Via header

A network trace of a client response looks like:

Frame: Number = 156, Captured Frame Length = 915, MediaType = ETHERNET
+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[02-00-4C-4F-4F-50],SourceAddress:[00-15-5D-21-17-0A]
+ Ipv4: Src = 10.1.0.12, Dest = 10.1.0.5, Next Protocol = TCP, Packet ID = 27779, Total IP Length = 901
+ Tcp: Flags=...AP..., SrcPort=9999, DstPort=46989, PayloadLen=861, Seq=3231455964 - 3231456825, Ack=892235420, Win=513 (scale factor 0x8) = 131328
- Http: Response, HTTP/1.1, Status: Ok, URL: /Shared%20Documents/
    ProtocolVersion: HTTP/1.1
    StatusCode: 200, Ok
    Reason: OK
    Cache-Control: private,max-age=0
    Allow: GET, POST, OPTIONS, HEAD, MKCOL, PUT, PROPFIND, PROPPATCH, DELETE, MOVE, COPY, GETLIB, LOCK, UNLOCK
    ContentLength: 0
    Expires: Mon, 30 Jan 2012 15:36:07 GMT
    Accept-Ranges: none
    Server: Microsoft-IIS/7.5
    SPRequestGuid: fb203b34-4b4b-4b65-8cf0-33dd90aa309d
    Set-Cookie: WSS_KeepSessionAuthenticated={58cbac57-f6eb-4c28-89ce-76505b69601b}; path=/
    X-SharePointHealthScore: 4
- WDVSEHeaderMsAuthorVia: MS-FP/4.0 -Microsoft FrontPage Server protocol version 4.0 enabled on the Web server.
     MsAuthorVia: MS-FP/4.0,DAV
    X-MSDAVEXT: 1
    DocumentManagementServer: Properties Schema;Source Control;Version History;
   X-MSFSSHTTP: 1.0
    DAV: 1,2
    Public-Extension: http://schemas.microsoft.com/repl-2
    Set-Cookie: WSS_KeepSessionAuthenticated={58cbac57-f6eb-4c28-89ce-76505b69601b}; path=/
    Persistent-Auth: true
    XPoweredBy: ASP.NET
    MicrosoftSharePointTeamServices: 14.0.0.6112
    Date: Tue, 14 Feb 2012 15:36:07 GMT
    HeaderEnd: CRLF

Another important piece to the above frame is the X-MSFSSHTTP header. Above you can see it equals 1.0 which means that the client will use File Synchronization via SOAP over HTTP protocol. This starts on # 12 below. It then looks at the MSAuthorVia to determine if the server supports the FrontPage Server Extensions. Since it does, FrontPage Server Extensions are used in this case.

If the X-MSFSSHTTP header is missing or less than 1.0, it just reverts to the MsAuthorVia to communicate over Webdav or Front Page Server Extensions.
In the above example it supports both Front Page Server Extensions and WebDav. Since both are provided, the determination of which protocol to use is based on the version of the Operating System and the Protocol Client that are in use.

What is used "WebDav or FPSE (Front Page Server Extensions).

Windows 7/Windows Vista with Office 2010/Office 2007 - WebDav is used.
Windows XP with Excel 2010\Excel 2007\PowerPoint 2010\PowerPoint 2007\Word 2010\Word 2007 - WebDav is used
Windows XP with any other Office 2010 or Office 2007 Application not listed above - FrontPage Server Extensions are used
Any Operating System with Office 2003 client - Front Page Server Extensions are used

Note: Again, this decision making is made when the X-MSFSSHTTP is missing or equates to less than 1.0. In my case, it equals 1.0 so it will always use File Synchronization via SOAP over HTTP protocol and FPSE (Front Page Server Extensions since I report it as supported.

Question: What are Front Page Server Extensions?

Answer: From: http://msdn.microsoft.com/en-us/library/ms442469.aspx

“Microsoft SharePoint Foundation and FrontPage Server Extensions 2002 from Microsoft are a set of programs that work in conjunction with Microsoft Internet Information Server (IIS) to support administering, authoring, and browsing a Web site. SharePoint Foundation and the FrontPage Server Extensions Remote Procedure Call (RPC) protocol govern the exchange of information between the client computer and the Web server running SharePoint Foundation. This communications protocol is layered on top of the same HTTP protocol that Web browsers use to interact with a Web server. The RPC protocol uses the HTTP POST request to send methods to SharePoint Foundation and the FrontPage Server Extensions. These requests enable the client to request documents, update Tasks lists, add new authors, and so on. The return values contain any requested information to the client in the form of an HTML document. When a client author opens a site from a Web server running SharePoint Foundation or FrontPage Server Extensions, information about the site, such as its hyperlink map, is downloaded to the client computer so that the client can display the information. However, the full set of pages and other files that comprise the site remain on the Web server.”

A list of FPRPC methods are available here: http://msdn.microsoft.com/en-us/library/ms443099.aspx

4. FPSE Request: The first FPSE frame the client sends is to request the Server version via HTTP Post against /_vti_bin/shtml.dll/_vti_rpc with the method name: server_version as documented here:

http://msdn.microsoft.com/en-us/library/ms460198.aspx

Frame: Number = 174, Captured Frame Length = 519, MediaType = ETHERNET
+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[00-15-5D-21-17-0A],SourceAddress:[02-00-4C-4F-4F-50]
+ Ipv4: Src = 10.1.0.5, Dest = 10.1.0.12, Next Protocol = TCP, Packet ID = 14380, Total IP Length = 505
+ Tcp: Flags=...AP..., SrcPort=46989, DstPort=9999, PayloadLen=465, Seq=892235759 - 892236224, Ack=3231457633, Win=32850 (scale factor 0x2) = 131400
- Http: Request, POST /_vti_bin/shtml.dll/_vti_rpc
    Command: POST
- URI: /_vti_bin/shtml.dll/_vti_rpc
     Location: /_vti_bin/shtml.dll/_vti_rpc
    ProtocolVersion: HTTP/1.1
    Date: Tue, 14 Feb 2012 15:36:07 GMT
    MIME-Version: 1.0
    UserAgent: MSFrontPage/14.0
    Host: wfe:9999
    Accept: auth/sicily
    ContentLength: 42
- ContentType: application/x-www-form-urlencoded
     MediaType: application/x-www-form-urlencoded
    X-Vermeer-Content-Type: application/x-www-form-urlencoded
    Connection: Keep-Alive
    Cache-Control: no-cache
+ Cookie: WSS_KeepSessionAuthenticated={58cbac57-f6eb-4c28-89ce-76505b69601b}
    HeaderEnd: CRLF
    payload: HttpContentType = application/x-www-form-urlencoded
- FrontPage: (Request) server_version:14.0.0.6009;
- FrontPageUrl: server_version:14.0.0.6009;
   - FPMethod: server_version:14.0.0.6009
      Method: server_version - Request the version of the server extensions in use on the Web server
    + ServerVersion: 14.0.0.6009
    End:

5. FPSE Response: The server responds with the server version information:

Frame: Number = 175, Captured Frame Length = 516, MediaType = ETHERNET
+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[02-00-4C-4F-4F-50],SourceAddress:[00-15-5D-21-17-0A]
+ Ipv4: Src = 10.1.0.12, Dest = 10.1.0.5, Next Protocol = TCP, Packet ID = 27781, Total IP Length = 502
+ Tcp: Flags=...AP..F, SrcPort=9999, DstPort=46989, PayloadLen=462, Seq=3231457633 - 3231458096, Ack=892236224, Win=510 (scale factor 0x8) = 130560
- Http: Response, HTTP/1.1, Status: Ok, URL: /_vti_bin/shtml.dll/_vti_rpc
 ProtocolVersion: HTTP/1.1
 StatusCode: 200, Ok
 Reason: OK
 Server: Microsoft-IIS/7.5
 Date: Tue, 14 Feb 2012 15:36:07 GMT
 Connection: close
+ ContentType: application/x-vermeer-rpc
 Set-Cookie: WSS_KeepSessionAuthenticated={58cbac57-f6eb-4c28-89ce-76505b69601b}; path=/
 HeaderEnd: CRLF
 payload: HttpContentType = application/x-vermeer-rpc
- FrontPage: (Response) server_version:14.0.0.6009;
- FrontPageHtml: 0x1
 FPSHtmlBegin: <html>
 + FPSHtmlHead: HtmlTitle = vermeer RPC packet
 - FPSHtmlBody: server_version:14.0.0.6009;
 FPSHtmlBodyBegin:
<body>
 - FPSHtmlBodyData: server_version:14.0.0.6009;
 + HtmlSign:

 + FPMethodHtml: server_version:14.0.0.6009
 + HtmlSign:

 + server version:
 + HtmlSign:

 + source control: 1
 FPSHtmlBodyEnd:
</body>
 FPSHtmlEnd:
</html>
 End:

6. FPSE Request: The client then fetches the URL of the file by submitting the FPRPC url_to_web method documented here:

http://msdn.microsoft.com/en-us/library/ms460544.aspx

Frame: Number = 187, Captured Frame Length = 143, MediaType = ETHERNET
+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[00-15-5D-21-17-0A],SourceAddress:[02-00-4C-4F-4F-50]
+ Ipv4: Src = 10.1.0.5, Dest = 10.1.0.12, Next Protocol = TCP, Packet ID = 14388, Total IP Length = 129
+ Tcp: Flags=...AP..., SrcPort=46990, DstPort=9999, PayloadLen=89, Seq=1159122353 - 1159122442, Ack=2274629607, Win=32649 (scale factor 0x2) = 130596
- Http: HTTP Payload, URL: /_vti_bin/shtml.dll/_vti_rpc
    payload: HttpContentType = application/x-www-form-urlencoded
- FrontPage: (Request) url_to_web_url:14.0.0.6009;Url = /Shared+Documents/Test8.docx;
- FrontPageUrl: url_to_web_url:14.0.0.6009;Url = /Shared+Documents/Test8.docx;
   - FPMethod: url_to_web_url:14.0.0.6009
      Method: url_to_web_url - Return the URL of the Web site to which a designated file belongs, including its subordinate Web if applicable
    + ServerVersion: 14.0.0.6009
   + Url: /Shared+Documents/Test8.docx
   + Flags: 0 - This parameter is ignored by the server but can be sent by the client and should equal 0
    End:

7. FPSE Response: The server responds with the URL location of the file:

Frame: Number = 189, Captured Frame Length = 450, MediaType = ETHERNET
+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[02-00-4C-4F-4F-50],SourceAddress:[00-15-5D-21-17-0A]
+ Ipv4: Src = 10.1.0.12, Dest = 10.1.0.5, Next Protocol = TCP, Packet ID = 27789, Total IP Length = 436
+ Tcp: Flags=...AP..F, SrcPort=9999, DstPort=46990, PayloadLen=396, Seq=2274629607 - 2274630004, Ack=1159122442, Win=513 (scale factor 0x8) = 131328
- Http: Response, HTTP/1.1, Status: Ok, URL: /_vti_bin/shtml.dll/_vti_rpc
 ProtocolVersion: HTTP/1.1
 StatusCode: 200, Ok
 Reason: OK
 Server: Microsoft-IIS/7.5
 Date: Tue, 14 Feb 2012 15:36:07 GMT
 Connection: close
+ ContentType: application/x-vermeer-rpc
 Set-Cookie: WSS_KeepSessionAuthenticated={58cbac57-f6eb-4c28-89ce-76505b69601b}; path=/
 HeaderEnd: CRLF
 payload: HttpContentType = application/x-vermeer-rpc
- FrontPage: (Response) url_to_web_url:14.0.0.6009;
- FrontPageHtml: 0x1
 FPSHtmlBegin: <html>
 - FPSHtmlHead: HtmlTitle = vermeer RPC packet
 FPSHtmlHeadBegin: <head>
 - FPSHtmlTitle: vermeer RPC packet
 FPSHtmlTitleBegin: <title>
 + FPSHtmlTitleData: vermeer RPC packet
 FPSHtmlTitleEnd: </title>
 FPSHtmlHeadEnd: </head>
 - FPSHtmlBody: url_to_web_url:14.0.0.6009;
 FPSHtmlBodyBegin:
<body>
 - FPSHtmlBodyData: url_to_web_url:14.0.0.6009;
 + HtmlSign:

 - FPMethodHtml: url_to_web_url:14.0.0.6009
 Method: url_to_web_url - Return the URL of the Web site to which a designated file belongs, including its subordinate Web if applicable
 + ServerVersion: 14.0.0.6009
 + HtmlSign:

 - webUrl: /
 char: /
 + HtmlSign:

 + fileUrl: Shared Documents/Test8.docx
 FPSHtmlBodyEnd:
</body>
 FPSHtmlEnd:
</html>
 End:

8. FPSE Request: The client then sends FPSE request meta-information for a Web-Site to the client application. It uses the open service method to facilitate this request.

From: http://msdn.microsoft.com/en-us/library/ms446353.aspx

Frame: Number = 201, Captured Frame Length = 113, MediaType = ETHERNET
+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[00-15-5D-21-17-0A],SourceAddress:[02-00-4C-4F-4F-50]
+ Ipv4: Src = 10.1.0.5, Dest = 10.1.0.12, Next Protocol = TCP, Packet ID = 14396, Total IP Length = 99
+ Tcp: Flags=...AP..., SrcPort=46991, DstPort=9999, PayloadLen=59, Seq=3991500678 - 3991500737, Ack=350150134, Win=32649 (scale factor 0x2) = 130596
- Http: HTTP Payload, URL: /_vti_bin/_vti_aut/author.dll
    payload: HttpContentType = application/x-www-form-urlencoded
- FrontPage: (Request) open_service:14.0.0.6009;ServiceName = /;
- FrontPageUrl: open_service:14.0.0.6009;ServiceName = /;
   - FPMethod: open_service:14.0.0.6009
      Method: open_service - Provides a Web's meta information to the client
    + ServerVersion: 14.0.0.6009
   - ServiceName: /
      FPByte: /
    End:

9. FPSE Response: The server provides the requested meta information of the site below:

Frame: Number = 203, Captured Frame Length = 1514, MediaType = ETHERNET
+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[02-00-4C-4F-4F-50],SourceAddress:[00-15-5D-21-17-0A]
+ Ipv4: Src = 10.1.0.12, Dest = 10.1.0.5, Next Protocol = TCP, Packet ID = 27802, Total IP Length = 1500
+ Tcp: Flags=...A...., SrcPort=9999, DstPort=46991, PayloadLen=1460, Seq=350150134 - 350151594, Ack=3991500737, Win=513 (scale factor 0x8) = 131328
- Http: Response, HTTP/1.1, Status: Ok, URL: /_vti_bin/_vti_aut/author.dll
 ProtocolVersion: HTTP/1.1
 StatusCode: 200, Ok
 Reason: OK
 Server: Microsoft-IIS/7.5
 Date: Tue, 14 Feb 2012 15:36:07 GMT
 Connection: close
+ ContentType: application/x-vermeer-rpc
 Set-Cookie: WSS_KeepSessionAuthenticated={58cbac57-f6eb-4c28-89ce-76505b69601b}; path=/
 HeaderEnd: CRLF
 payload: HttpContentType = application/x-vermeer-rpc
- FrontPage: (Response) open_service:14.0.0.6009;
- FrontPageHtml: 0x1
 FPSHtmlBegin: <html>
 - FPSHtmlHead: HtmlTitle = vermeer RPC packet
 FPSHtmlHeadBegin: <head>
 - FPSHtmlTitle: vermeer RPC packet
 FPSHtmlTitleBegin: <title>
 + FPSHtmlTitleData: vermeer RPC packet
 FPSHtmlTitleEnd: </title>
 FPSHtmlHeadEnd: </head>
 - FPSHtmlBody: open_service:14.0.0.6009;
 FPSHtmlBodyBegin:
<body>
 - FPSHtmlBodyData: open_service:14.0.0.6009;
 + HtmlSign:

 - FPMethodHtml: open_service:14.0.0.6009
 Method: open_service - Provides a Web's meta information to the client
 + ServerVersion: 14.0.0.6009
 + HtmlSign:

 - service:
 - ListInfo: 0x1 Members
 ListInfoHtmlBegin: [
 InfoBeginHtml:
 - InfoData: ServiceName = ;
 + service_name:
 - MetaInfo:
 MetaInfoHtmlBegin: [
 InfoBeginHtml:
 + MetaInfoData: vti_defaultlanguage = en-us (Type = String and Write Enable)
 + MetaInfoData: ratings_emptyiconurl = /_layouts/Images/RatingsEmpty.png (Type = String and Write Enable)
 + MetaInfoData: vti_servercharsets = windows-1257 iso-8859-5 big5 windows-1252 iso-8859-9 windows-874 utf-8 windows-1256 gb2312 iso-8859-4 windows-1251 ks_c_5601-1987 iso-8859-6 gb18030 iso-2022-jp koi8-r iso-8859-1 iso-8859-8 windows-1253 windows-1258 windows-
 + MetaInfoData: vti_toolpaneurl = http://wfe:9999/_layouts/toolpane.aspx (Type = String and Ignore)
 + MetaInfoData: vti_assemblyversion = Microsoft.SharePoint, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c (Type = String and Ignore)
 + MetaInfoData: vti_customuploadpage = /_layouts/UploadEx.aspx (Type = String and Write Enable)
 + MetaInfoData: vti_hasonetlayoutfiles = true (Type = Boolean and Read Only)
 + MetaInfoData: vti_associategroups = 7;3;3;6;5 (Type = String and Write Enable)
 + MetaInfoData: enabledhelpcollections = VGSEndUser;#FastEndUser;#SQLWSSAddIn (Type = String and Write Enable)
 + MetaInfoData: mossMDNHints_b80ea5a7-c89d-4dde-aaea-59857ab62704 = 0 (Type = String and Write Enable)
 + MetaInfoData: vti_casesensitiveurls = 0 (Type = Int and Ignore)
 + MetaInfoData: vti_masterurl = /_catalogs/masterpage/v4.master (Type = String and Ignore)
 + MetaInfoData: disabledhelpcol = /_catalogs/masterpage/v4.master (Type = )

10. FPSE Request: The client request the document metainfo by running method get_docs_metainfo against the targeted document.

http://msdn.microsoft.com/en-us/library/ms457496.aspx

Frame: Number = 216, Captured Frame Length = 257, MediaType = ETHERNET
+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[00-15-5D-21-17-0A],SourceAddress:[02-00-4C-4F-4F-50]
+ Ipv4: Src = 10.1.0.5, Dest = 10.1.0.12, Next Protocol = TCP, Packet ID = 14404, Total IP Length = 243
+ Tcp: Flags=...AP..., SrcPort=46992, DstPort=9999, PayloadLen=203, Seq=3889723891 - 3889724094, Ack=642547811, Win=32714 (scale factor 0x2) = 130856
- Http: HTTP Payload, URL: /_vti_bin/_vti_aut/author.dll
    payload: HttpContentType = application/x-www-form-urlencoded
- FrontPage: (Request) get_docs_metainfo:14.0.0.6009;
- FrontPageUrl: get_docs_metainfo:14.0.0.6009;
   - FPMethod: get_docs_metainfo:14.0.0.6009
     Method: get_docs_metainfo - Retrieve meta information for the files in the current Web
    + ServerVersion: 14.0.0.6009
   - Url_List: http://wfe:9999/Shared Documents/Test8.docx;http://wfe:9999/Shared Documents;
    - ListInfo: http://wfe:9999/Shared Documents/Test8.docx;http://wfe:9999/Shared Documents;
       ListInfoUrlBegin: [
     + InfoListData: http://wfe:9999/Shared Documents/Test8.docx
     + InfoListData: http://wfe:9999/Shared Documents
       ListInfoUrlEnd: ]
      Terminal: &
   + ListHiddenDocs: false
   + ListLinkInfo: false
    End:

11. FPSE Response: The server responds with the metainfo for the document.

Frame: Number = 218, Captured Frame Length = 1514, MediaType = ETHERNET
+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[02-00-4C-4F-4F-50],SourceAddress:[00-15-5D-21-17-0A]
+ Ipv4: Src = 10.1.0.12, Dest = 10.1.0.5, Next Protocol = TCP, Packet ID = 27815, Total IP Length = 1500
+ Tcp: Flags=...A...., SrcPort=9999, DstPort=46992, PayloadLen=1460, Seq=642547811 - 642549271, Ack=3889724094, Win=508 (scale factor 0x8) = 130048
- Http: Response, HTTP/1.1, Status: Ok, URL: /_vti_bin/_vti_aut/author.dll
 ProtocolVersion: HTTP/1.1
 StatusCode: 200, Ok
 Reason: OK
 Server: Microsoft-IIS/7.5
 Date: Tue, 14 Feb 2012 15:36:08 GMT
 Connection: close
+ ContentType: application/x-vermeer-rpc
 Set-Cookie: WSS_KeepSessionAuthenticated={58cbac57-f6eb-4c28-89ce-76505b69601b}; path=/
 HeaderEnd: CRLF
 payload: HttpContentType = application/x-vermeer-rpc
- FrontPage: (Response) get_docs_metainfo:14.0.0.6009;
- FrontPageHtml: 0x1
 FPSHtmlBegin: <html>
 - FPSHtmlHead: HtmlTitle = vermeer RPC packet
 FPSHtmlHeadBegin: <head>
 - FPSHtmlTitle: vermeer RPC packet
 FPSHtmlTitleBegin: <title>
 + FPSHtmlTitleData: vermeer RPC packet
 FPSHtmlTitleEnd: </title>
 FPSHtmlHeadEnd: </head>
 - FPSHtmlBody: get_docs_metainfo:14.0.0.6009;
 FPSHtmlBodyBegin:
<body>
 - FPSHtmlBodyData: get_docs_metainfo:14.0.0.6009;
 + HtmlSign:

 - FPMethodHtml: get_docs_metainfo:14.0.0.6009
 Method: get_docs_metainfo - Retrieve meta information for the files in the current Web
 + ServerVersion: 14.0.0.6009
 + HtmlSign:

 - DocumentList:
 TopLevelstart:
<ul>
 - ListInfo: 0x1 Members
 ListInfoHtmlBegin: [
 InfoBeginHtml:
 - InfoData: DocumentName = Shared Documents/Test8.docx;
 + DocumetName: Shared Documents/Test8.docx
 - MetaInfo:
 MetaInfoHtmlBegin: [
 InfoBeginHtml:
 + MetaInfoData: Subject = (Type = Unknown Type)
 + MetaInfoData: vti_rtag = rt:B33158C8-8CF0-4BBF-8001-3028AD6B5350@00000000001 (Type = String and Write Enable)
 + MetaInfoData: vti_etag = "{B33158C8-8CF0-4BBF-8001-3028AD6B5350},1" (Type = String and Write Enable)
 + MetaInfoData: vti_thumbnailexists = false (Type = Boolean and Write Enable)
 + MetaInfoData: vti_parserversion = 14.0.0.6112 (Type = String and Read Only)
 + MetaInfoData: vti_timecreated = 14 Feb 2012 03:33:44 -0000 (Type = Time and Read Only)
 + MetaInfoData: vti_folderitemcount = 0 (Type = Int and Read Only)
 + MetaInfoData: _Category = (Type = Unknown Type)
 + MetaInfoData: vti_stickycachedpluggableparserprops = Subject Keywords _Status vti_title _Author _Category ContentType _Comments (Type = String Vector and Ignore)
 + MetaInfoData: vti_canmaybeedit = true (Type = Boolean and Ignore)
 + MetaInfoData: _Comments = (Type = Unknown Type)
 + MetaInfoData: vti_author = SHAREPOINT\system (Type = String and Read Only)
 + MetaInfoData: vti_sourcecontrolversion = V1.0 (Type = String and Read Only)
 + MetaInfoData: vti_sourcecontrolcookie = fp_internal (Type = String and Read Only)
 + MetaInfoData: vti_previewexists = false (Type = Boolean and Write Enable)
 + MetaInfoData: vti_level = 1 (Type = Int and Read Only)
 + MetaInfoData: Keywords = (Type = Unknown Type)
 + MetaInfoData: _Status = (Type = Unknown Type)
 + MetaInfoData: vti_modifiedby = SHAREPOINT\system (Type = String and Read Only)
 + MetaInfoData: vti_filesize = 18411 (Type = Int and Read Only)
 + MetaInfoData: vti_foldersubfolderitemcount = 0 (Type = Int and Read Only)
 + MetaInfoData: vti_title = (Type = Unknown Type)
 + MetaInfoData: _Author = Russ Maxwell (Type = String and Write Enable)
 + MetaInfoData: ContentType = (Type = Unknown Type)
 + MetaInfoData: ContentTypeId = 0x010100190C2FB49152074EBE0F739178D89BF8
<li (Type = String and Write Enable)

12. HTTP – SOAP Request: Client submits a POST to /_vti_bin/cellstoreage.svc/CellStorageService

This service allows client computers to synchronize changes made to shared files that are stored on a server. In this case, it’s used to download the file from the server to the client. This is also referred to as the “File Synchronization via SOAP over HTTP Protocol”

http://msdn.microsoft.com/en-us/library/websvccellstorage.aspx

Frame: Number = 232, Captured Frame Length = 757, MediaType = ETHERNET
+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[00-15-5D-21-17-0A],SourceAddress:[02-00-4C-4F-4F-50]
+ Ipv4: Src = 10.1.0.5, Dest = 10.1.0.12, Next Protocol = TCP, Packet ID = 14409, Total IP Length = 743
+ Tcp: Flags=...AP..., SrcPort=46993, DstPort=9999, PayloadLen=703, Seq=3699146994 - 3699147697, Ack=2981864978, Win=513 (scale factor 0x8) = 131328
- Http: Request, POST /_vti_bin/cellstorage.svc/CellStorageService , Using NTLM Authorization
 Command: POST
- URI: /_vti_bin/cellstorage.svc/CellStorageService
 Location: /_vti_bin/cellstorage.svc/CellStorageService
 ProtocolVersion: HTTP/1.1
 Cache-Control: no-cache
 Connection: Keep-Alive
 Pragma: no-cache
- ContentType: multipart/related; type="application/xop+xml"; boundary="urn:uuid:8adafd52-a33a-4665-b535-d99d195335d6"; start="<69213406-6aa3-4ffb-a2da-c6fae4d07276@tempuri.org>"; start-Info="text/xml; charset=utf-8"
 + MediaType: multipart/related; type="application/xop+xml"; boundary="urn:uuid:8adafd52-a33a-4665-b535-d99d195335d6"; start="<69213406-6aa3-4ffb-a2da-c6fae4d07276@tempuri.org>"; start-Info="text/xml; charset=utf-8"
+ Cookie: WSS_KeepSessionAuthenticated={58cbac57-f6eb-4c28-89ce-76505b69601b}
 UserAgent: Microsoft Office Word 2010 (14.0.6023) Windows NT 6.1
- SOAPAction: "http://schemas.microsoft.com/sharepoint/soap/ICellStorages/ExecuteCellStorageRequest"
 Value: http://schemas.microsoft.com/sharepoint/soap/ICellStorages/ExecuteCellStorageRequest
 ContentLength: 0
 Host: wfe:9999
+ Authorization: NTLM
 HeaderEnd: CRLF

13. HTTP Response from Server

Frame: Number = 238, Captured Frame Length = 1425, MediaType = ETHERNET
+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[02-00-4C-4F-4F-50],SourceAddress:[00-15-5D-21-17-0A]
+ Ipv4: Src = 10.1.0.12, Dest = 10.1.0.5, Next Protocol = TCP, Packet ID = 27842, Total IP Length = 1411
+ Tcp: Flags=...AP..., SrcPort=9999, DstPort=46993, PayloadLen=1371, Seq=2981865521 - 2981866892, Ack=3699151391, Win=509 (scale factor 0x8) = 130304
- Http: Response, HTTP/1.1, Status: Ok, URL: /_vti_bin/cellstorage.svc/CellStorageService
 ProtocolVersion: HTTP/1.1
 StatusCode: 200, Ok
 Reason: OK
 Cache-Control: private
 TransferEncoding: chunked
+ ContentType: multipart/related; type="application/xop+xml";start="<http://tempuri.org/0>";boundary="uuid:973cbe86-3867-47b7-bf03-927ddd6ba2ed+id=3";start-info="text/xml"
 Server: Microsoft-IIS/7.5
 SPRequestGuid: 7f5c207a-6f55-4049-8e8c-7e5d9b8a067b
 Set-Cookie: WSS_KeepSessionAuthenticated={58cbac57-f6eb-4c28-89ce-76505b69601b}; path=/
 X-SharePointHealthScore: 4
 MIME-Version: 1.0
 XAspNetVersion: 2.0.50727
 Persistent-Auth: true
 XPoweredBy: ASP.NET
 MicrosoftSharePointTeamServices: 14.0.0.6112
 Date: Tue, 14 Feb 2012 15:36:08 GMT
 HeaderEnd: CRLF
- chunkSize: 747
 Size: 747
 ChunkPayload: HttpContentType = multipart/related; type="application/xop+xml";start="<http://tempuri.org/0>";boundary="uuid:973cbe86-3867-47b7-bf03-927ddd6ba2ed+id=3";start-info="text/xml"
 FooterEnd: CRLF
+ Mime: MediaType = application/xop+xml
- Soap: xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"
- Envelope: <s:Envelope>
 - STag: <s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
 - Tag: <s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
 - TagName: s:Envelope
 NamespacePrefix: s
 Name: Envelope
 + Attributes: xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"
 - Body: <s:Body>
 - STag: <s:Body>
 - Tag: <s:Body>
 - TagName: s:Body
 NamespacePrefix: s
 Name: Body
 - Node: XmlElement:<ResponseVersion>
 - EmptyElement: <ResponseVersion Version="2" MinorVersion="0" xmlns="http://schemas.microsoft.com/sharepoint/soap/"/>
 - Tag: <ResponseVersion Version="2" MinorVersion="0" xmlns="http://schemas.microsoft.com/sharepoint/soap/"/>
 - TagName: ResponseVersion
 Name: ResponseVersion
 + Attributes: Version="2" MinorVersion="0" xmlns="http://schemas.microsoft.com/sharepoint/soap/"
 - Node: XmlElement:<ResponseCollection>
 - STag: <ResponseCollection WebUrl="http://wfe:9999" xmlns="http://schemas.microsoft.com/sharepoint/soap/">
 - Tag: <ResponseCollection WebUrl="http://wfe:9999" xmlns="http://schemas.microsoft.com/sharepoint/soap/">
 - TagName: ResponseCollection
 Name: ResponseCollection
 + Attributes: WebUrl="http://wfe:9999" xmlns="http://schemas.microsoft.com/sharepoint/soap/"
 - Element: XmlElement:<Response>
 - STag: <Response Url="http://wfe:9999/Shared%20Documents/Test8.docx" RequestToken="1" HealthScore="4">
 - Tag: <Response Url="http://wfe:9999/Shared%20Documents/Test8.docx" RequestToken="1" HealthScore="4">
 - TagName: Response
 Name: Response
 + Attributes: Url="http://wfe:9999/Shared%20Documents/Test8.docx" RequestToken="1" HealthScore="4"
 - Element: XmlElement:<SubResponse>
 - STag: <SubResponse SubRequestToken="1" ErrorCode="Success" HResult="0">
 - Tag: <SubResponse SubRequestToken="1" ErrorCode="Success" HResult="0">
 - TagName: SubResponse
 Name: SubResponse
 + Attributes: SubRequestToken="1" ErrorCode="Success" HResult="0"
 - Element: XmlElement:<SubResponseData>
 - EmptyElement: <SubResponseData LockType="SchemaLock" CoauthStatus="Alone" TransitionID="b33158c8-8cf0-4bbf-8001-3028ad6b5350"/>
 - Tag: <SubResponseData LockType="SchemaLock" CoauthStatus="Alone" TransitionID="b33158c8-8cf0-4bbf-8001-3028ad6b5350"/>
 - TagName: SubResponseData
 Name: SubResponseData
 + Attributes: LockType="SchemaLock" CoauthStatus="Alone" TransitionID="b33158c8-8cf0-4bbf-8001-3028ad6b5350"
 - ETag: </SubResponse>
 - Tag: </SubResponse>
 - TagName: SubResponse
 Name: SubResponse
 ETag:

The subsequent frames are all coming from the server via HTTP Payload packets from cellstorage.svc. I believe this is when it’s streaming the file to the client and not easy to read because it does so in chunks in BLOB format.

For Example:

      Frame: Number = 242, Captured Frame Length = 439, MediaType = ETHERNET
+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[02-00-4C-4F-4F-50],SourceAddress:[00-15-5D-21-17-0A]
+ Ipv4: Src = 10.1.0.12, Dest = 10.1.0.5, Next Protocol = TCP, Packet ID = 27855, Total IP Length = 425
+ Tcp: Flags=...AP..., SrcPort=9999, DstPort=46993, PayloadLen=385, Seq=2981867428 - 2981867813, Ack=3699151391, Win=509 (scale factor 0x8) = 130304
- Http: HTTP Payload, URL: /_vti_bin/cellstorage.svc/CellStorageService
- chunkSize: 378
     Size: 378
   ChunkPayloadContinuation: Binary Large Object (378 Bytes)
        FooterEnd: CRLF

14. HTTP Request: The client submits a Post to webs.asmx which is a request for a URL of the parent site for the specified page URL:

http://msdn.microsoft.com/en-us/library/websvcwebs.webs.weburlfrompageurl.aspx

Frame: Number = 315, Captured Frame Length = 1119, MediaType = ETHERNET
+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[00-15-5D-21-17-0A],SourceAddress:[02-00-4C-4F-4F-50]
+ Ipv4: Src = 10.1.0.5, Dest = 10.1.0.12, Next Protocol = TCP, Packet ID = 14427, Total IP Length = 1105
+ Tcp: Flags=...AP..., SrcPort=46994, DstPort=9999, PayloadLen=1065, Seq=3401034132 - 3401035197, Ack=1507241640, Win=32649 (scale factor 0x2) = 130596
- Http: Request, POST /_vti_bin/webs.asmx , Using NTLM Authorization
    Command: POST
- URI: /_vti_bin/webs.asmx
     Location: /_vti_bin/webs.asmx
    ProtocolVersion: HTTP/1.1
- ContentType: text/xml; charset=utf-8
   + MediaType: text/xml; charset=utf-8
- SOAPAction: http://schemas.microsoft.com/sharepoint/soap/WebUrlFromPageUrl
     DefaultValue: http://schemas.microsoft.com/sharepoint/soap/WebUrlFromPageUrl
    X-Office-Version: 14.0.6106
    UserAgent: Microsoft Office/14.0 (Windows NT 6.1; Microsoft Word 14.0.6106; Pro)
    Host: wfe:9999
    ContentLength: 309
    Connection: Keep-Alive
    Cache-Control: no-cache
+ Cookie: WSS_KeepSessionAuthenticated={58cbac57-f6eb-4c28-89ce-76505b69601b}
+ Authorization: NTLM
    HeaderEnd: CRLF

15. SOAP Request: The client includes the following SOAP request in the subsequent frame:

Frame: Number = 316, Captured Frame Length = 363, MediaType = ETHERNET
+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[00-15-5D-21-17-0A],SourceAddress:[02-00-4C-4F-4F-50]
+ Ipv4: Src = 10.1.0.5, Dest = 10.1.0.12, Next Protocol = TCP, Packet ID = 14428, Total IP Length = 349
+ Tcp: Flags=...AP..., SrcPort=46994, DstPort=9999, PayloadLen=309, Seq=3401035197 - 3401035506, Ack=1507241640, Win=32649 (scale factor 0x2) = 130596
- Http: HTTP Payload, URL: /_vti_bin/webs.asmx
- payload: HttpContentType = text/xml; charset=utf-8
 - Xml: version="1.0" encoding="utf-8"
 XmlPropertiesInitiator:
 - XMLDecl: XmlElement:<{xml directive}>
 - Directive: <?xml version="1.0" encoding="utf-8"?>
 - Tag: <?xml version="1.0" encoding="utf-8"?>
 - TagName: xml
 Name: xml
 + Attributes: version="1.0" encoding="utf-8"
- Soap: xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
- Envelope: <soap:Envelope>
 - STag: <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
 - Tag: <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
 - TagName: soap:Envelope
 NamespacePrefix: soap
 Name: Envelope
 + Attributes: xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
 - Body: <soap:Body>
 - STag: <soap:Body>
 - Tag: <soap:Body>
 - TagName: soap:Body
 NamespacePrefix: soap
 Name: Body
 - Node: XmlElement:<WebUrlFromPageUrl>
 - STag: <WebUrlFromPageUrl xmlns="http://schemas.microsoft.com/sharepoint/soap/">
 - Tag: <WebUrlFromPageUrl xmlns="http://schemas.microsoft.com/sharepoint/soap/">
 - TagName: WebUrlFromPageUrl
 Name: WebUrlFromPageUrl
 + Attributes: xmlns="http://schemas.microsoft.com/sharepoint/soap/"
 - Element: XmlElement:<pageUrl> - http://wfe:9999/Shared%20Documents/Test8.docx
 - STag: <pageUrl>
 - Tag: <pageUrl>
 - TagName: pageUrl
 Name: pageUrl
 Content: http://wfe:9999/Shared%20Documents/Test8.docx
 - ETag: </pageUrl>
 - Tag: </pageUrl>
 - TagName: pageUrl
 Name: pageUrl
 - ETag: </WebUrlFromPageUrl>
 - Tag: </WebUrlFromPageUrl>
 - TagName: WebUrlFromPageUrl
 Name: WebUrlFromPageUrl
 - ETag: </soap:Body>
 - Tag: </soap:Body>
 - TagName: soap:Body
 NamespacePrefix: soap
 Name: Body
 - ETag: </soap:Envelope>
 - Tag: </soap:Envelope>
 - TagName: soap:Envelope
 NamespacePrefix: soap
 Name: Envelope

16. SOAP Response: Server responds to the above SOAP request with the URL of the parent site

Frame: Number = 318, Captured Frame Length = 944, MediaType = ETHERNET
+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[02-00-4C-4F-4F-50],SourceAddress:[00-15-5D-21-17-0A]
+ Ipv4: Src = 10.1.0.12, Dest = 10.1.0.5, Next Protocol = TCP, Packet ID = 27919, Total IP Length = 930
+ Tcp: Flags=...AP..., SrcPort=9999, DstPort=46994, PayloadLen=890, Seq=1507241640 - 1507242530, Ack=3401035506, Win=513 (scale factor 0x8) = 131328
- Http: Response, HTTP/1.1, Status: Ok, URL: /_vti_bin/webs.asmx
 ProtocolVersion: HTTP/1.1
 StatusCode: 200, Ok
 Reason: OK
 Cache-Control: private, max-age=0
+ ContentType: text/xml; charset=utf-8
 Server: Microsoft-IIS/7.5
 SPRequestGuid: 20f0848a-331c-49f5-83f8-1b8da3e6a6f6
 Set-Cookie: WSS_KeepSessionAuthenticated={58cbac57-f6eb-4c28-89ce-76505b69601b}; path=/
 X-SharePointHealthScore: 4
 XAspNetVersion: 2.0.50727
 Persistent-Auth: true
 XPoweredBy: ASP.NET
 MicrosoftSharePointTeamServices: 14.0.0.6112
 Date: Tue, 14 Feb 2012 15:36:09 GMT
 ContentLength: 421
 HeaderEnd: CRLF
- payload: HttpContentType = text/xml; charset=utf-8
 - Xml: version="1.0" encoding="utf-8"
 XmlPropertiesInitiator:
 - XMLDecl: XmlElement:<{xml directive}>
 - Directive: <?xml version="1.0" encoding="utf-8"?>
 - Tag: <?xml version="1.0" encoding="utf-8"?>
 - TagName: xml
 Name: xml
 + Attributes: version="1.0" encoding="utf-8"
- Soap: xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"
- Envelope: <soap:Envelope>
 - STag: <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
 - Tag: <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
 - TagName: soap:Envelope
 NamespacePrefix: soap
 Name: Envelope
 + Attributes: xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"
 - Body: <soap:Body>
 - STag: <soap:Body>
 - Tag: <soap:Body>
 - TagName: soap:Body
 NamespacePrefix: soap
 Name: Body
 - Node: XmlElement:<WebUrlFromPageUrlResponse>
 - STag: <WebUrlFromPageUrlResponse xmlns="http://schemas.microsoft.com/sharepoint/soap/">
 - Tag: <WebUrlFromPageUrlResponse xmlns="http://schemas.microsoft.com/sharepoint/soap/">
 - TagName: WebUrlFromPageUrlResponse
 Name: WebUrlFromPageUrlResponse
 + Attributes: xmlns="http://schemas.microsoft.com/sharepoint/soap/"
 - Element: XmlElement:<WebUrlFromPageUrlResult> - http://wfe:9999
 - STag: <WebUrlFromPageUrlResult>
 - Tag: <WebUrlFromPageUrlResult>
 - TagName: WebUrlFromPageUrlResult
 Name: WebUrlFromPageUrlResult
 Content: http://wfe:9999
 - ETag: </WebUrlFromPageUrlResult>
 - Tag: </WebUrlFromPageUrlResult>
 - TagName: WebUrlFromPageUrlResult
 Name: WebUrlFromPageUrlResult
 - ETag: </WebUrlFromPageUrlResponse>
 - Tag: </WebUrlFromPageUrlResponse>
 - TagName: WebUrlFromPageUrlResponse
 Name: WebUrlFromPageUrlResponse
 - ETag: </soap:Body>
 - Tag: </soap:Body>
 - TagName: soap:Body
 NamespacePrefix: soap
 Name: Body
 - ETag: </soap:Envelope>
 - Tag: </soap:Envelope>
 - TagName: soap:Envelope
 NamespacePrefix: soap
 Name: Envelope

17. SOAP Request: The last request from client is a SOAP request to /_vti_bin/workflow.asmx and calls GetWorkflowDataForItem which pulls the workflow data for a specific user for a given item.

http://msdn.microsoft.com/en-us/library/workflow.workflow.getworkflowdataforitem(v=office.12).aspx

Frame: Number = 319, Captured Frame Length = 823, MediaType = ETHERNET
+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[00-15-5D-21-17-0A],SourceAddress:[02-00-4C-4F-4F-50]
+ Ipv4: Src = 10.1.0.5, Dest = 10.1.0.12, Next Protocol = TCP, Packet ID = 14429, Total IP Length = 809
+ Tcp: Flags=...AP..., SrcPort=46994, DstPort=9999, PayloadLen=769, Seq=3401035506 - 3401036275, Ack=1507242530, Win=32850 (scale factor 0x2) = 131400
- Http: Request, POST /_vti_bin/workflow.asmx
 Command: POST
- URI: /_vti_bin/workflow.asmx
 Location: /_vti_bin/workflow.asmx
 ProtocolVersion: HTTP/1.1
- ContentType: text/xml; charset=utf-8
 + MediaType: text/xml; charset=utf-8
- SOAPAction: http://schemas.microsoft.com/sharepoint/soap/workflow/GetWorkflowDataForItem
 DefaultValue: http://schemas.microsoft.com/sharepoint/soap/workflow/GetWorkflowDataForItem
 X-Office-Version: 14.0.6106
 UserAgent: Microsoft Office/14.0 (Windows NT 6.1; Microsoft Word 14.0.6106; Pro)
 Host: wfe:9999
 ContentLength: 324
 Connection: Keep-Alive
 Cache-Control: no-cache
+ Cookie: WSS_KeepSessionAuthenticated={58cbac57-f6eb-4c28-89ce-76505b69601b}
 HeaderEnd: CRLF
- payload: HttpContentType = text/xml; charset=utf-8
 - Xml: version="1.0" encoding="utf-8"
 XmlPropertiesInitiator:
 - XMLDecl: XmlElement:<{xml directive}>
 - Directive: <?xml version="1.0" encoding="utf-8"?>
 - Tag: <?xml version="1.0" encoding="utf-8"?>
 - TagName: xml
 Name: xml
 + Attributes: version="1.0" encoding="utf-8"
- Soap: xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
- Envelope: <soap:Envelope>
 - STag: <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
 - Tag: <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
 - TagName: soap:Envelope
 NamespacePrefix: soap
 Name: Envelope
 + Attributes: xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
 - Body: <soap:Body>
 - STag: <soap:Body>
 - Tag: <soap:Body>
 - TagName: soap:Body
 NamespacePrefix: soap
 Name: Body
 - Node: XmlElement:<GetWorkflowDataForItem>
 - STag: <GetWorkflowDataForItem xmlns="http://schemas.microsoft.com/sharepoint/soap/workflow/">
 - Tag: <GetWorkflowDataForItem xmlns="http://schemas.microsoft.com/sharepoint/soap/workflow/">
 - TagName: GetWorkflowDataForItem
 Name: GetWorkflowDataForItem
 + Attributes: xmlns="http://schemas.microsoft.com/sharepoint/soap/workflow/"
 - Element: XmlElement:<item> - http://wfe:9999/Shared%20Documents/Test8.docx
 - STag: <item>
 - Tag: <item>
 - TagName: item
 Name: item
 Content: http://wfe:9999/Shared%20Documents/Test8.docx
 - ETag: </item>
 - Tag: </item>
 - TagName: item
 Name: item
 - ETag: </GetWorkflowDataForItem>
 - Tag: </GetWorkflowDataForItem>
 - TagName: GetWorkflowDataForItem
 Name: GetWorkflowDataForItem
 - ETag: </soap:Body>
 - Tag: </soap:Body>
 - TagName: soap:Body
 NamespacePrefix: soap
 Name: Body
 - ETag: </soap:Envelope>
 - Tag: </soap:Envelope>
 - TagName: soap:Envelope
 NamespacePrefix: soap
 Name: Envelope

18. SOAP Response: Server responds and provides workflow data per client request

Frame: Number = 320, Captured Frame Length = 1273, MediaType = ETHERNET
+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[02-00-4C-4F-4F-50],SourceAddress:[00-15-5D-21-17-0A]
+ Ipv4: Src = 10.1.0.12, Dest = 10.1.0.5, Next Protocol = TCP, Packet ID = 27933, Total IP Length = 1259
+ Tcp: Flags=...AP..., SrcPort=9999, DstPort=46994, PayloadLen=1219, Seq=1507242530 - 1507243749, Ack=3401036275, Win=510 (scale factor 0x8) = 130560
- Http: Response, HTTP/1.1, Status: Ok, URL: /_vti_bin/workflow.asmx
 ProtocolVersion: HTTP/1.1
 StatusCode: 200, Ok
 Reason: OK
 Cache-Control: private, max-age=0
+ ContentType: text/xml; charset=utf-8
 Server: Microsoft-IIS/7.5
 SPRequestGuid: 31528070-4810-4c8a-b228-c8594f0cce4b
 Set-Cookie: WSS_KeepSessionAuthenticated={58cbac57-f6eb-4c28-89ce-76505b69601b}; path=/
 X-SharePointHealthScore: 4
 XAspNetVersion: 2.0.50727
 XPoweredBy: ASP.NET
 MicrosoftSharePointTeamServices: 14.0.0.6112
 Date: Tue, 14 Feb 2012 15:36:09 GMT
 ContentLength: 773
 HeaderEnd: CRLF
- payload: HttpContentType = text/xml; charset=utf-8
 - Xml: version="1.0" encoding="utf-8"
 XmlPropertiesInitiator:
 - XMLDecl: XmlElement:<{xml directive}>
 - Directive: <?xml version="1.0" encoding="utf-8"?>
 - Tag: <?xml version="1.0" encoding="utf-8"?>
 - TagName: xml
 Name: xml
 + Attributes: version="1.0" encoding="utf-8"
- Soap: xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"
- Envelope: <soap:Envelope>
 - STag: <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
 - Tag: <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
 - TagName: soap:Envelope
 NamespacePrefix: soap
 Name: Envelope
 + Attributes: xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"
 - Body: <soap:Body>
 - STag: <soap:Body>
 - Tag: <soap:Body>
 - TagName: soap:Body
 NamespacePrefix: soap
 Name: Body
 - Node: XmlElement:<GetWorkflowDataForItemResponse>
 - STag: <GetWorkflowDataForItemResponse xmlns="http://schemas.microsoft.com/sharepoint/soap/workflow/">
 - Tag: <GetWorkflowDataForItemResponse xmlns="http://schemas.microsoft.com/sharepoint/soap/workflow/">
 - TagName: GetWorkflowDataForItemResponse
 Name: GetWorkflowDataForItemResponse
 + Attributes: xmlns="http://schemas.microsoft.com/sharepoint/soap/workflow/"
 - Element: XmlElement:<GetWorkflowDataForItemResult>
 - STag: <GetWorkflowDataForItemResult>
 - Tag: <GetWorkflowDataForItemResult>
 - TagName: GetWorkflowDataForItemResult
 Name: GetWorkflowDataForItemResult
 - Element: XmlElement:<WorkflowData>
 - STag: <WorkflowData>
 - Tag: <WorkflowData>
 - TagName: WorkflowData
 Name: WorkflowData
 - Element: XmlElement:<ToDoData>
 - STag: <ToDoData>
 - Tag: <ToDoData>
 - TagName: ToDoData
 Name: ToDoData
 - ETag: </ToDoData>
 - Tag: </ToDoData>
 - TagName: ToDoData
 Name: ToDoData
 - Element: XmlElement:<TemplateData>
 - STag: <TemplateData>
 - Tag: <TemplateData>
 - TagName: TemplateData
 Name: TemplateData
 - Element: XmlElement:<Web>
 - EmptyElement: <Web Title="HiAgain" Url="http://wfe:9999" />
 - Tag: <Web Title="HiAgain" Url="http://wfe:9999" />
 - TagName: Web
 Name: Web
 + Attributes: Title="HiAgain" Url="http://wfe:9999"
 - Element: XmlElement:<List>
 - EmptyElement: <List Title="Shared Documents" Url="http://wfe:9999/Shared Documents" />
 - Tag: <List Title="Shared Documents" Url="http://wfe:9999/Shared Documents" />
 - TagName: List
 Name: List
 + Attributes: Title="Shared Documents" Url="http://wfe:9999/Shared Documents"
 - Element: XmlElement:<WorkflowTemplates>
 - STag: <WorkflowTemplates>
 - Tag: <WorkflowTemplates>
 - TagName: WorkflowTemplates
 Name: WorkflowTemplates
 - ETag: </WorkflowTemplates>
 - Tag: </WorkflowTemplates>
 - TagName: WorkflowTemplates
 Name: WorkflowTemplates
 - ETag: </TemplateData>
 - Tag: </TemplateData>
 - TagName: TemplateData
 Name: TemplateData
 - Element: XmlElement:<ActiveWorkflowsData>
 - STag: <ActiveWorkflowsData>
 - Tag: <ActiveWorkflowsData>
 - TagName: ActiveWorkflowsData
 Name: ActiveWorkflowsData
 - Element: XmlElement:<Workflows>
 - STag: <Workflows>
 - Tag: <Workflows>
 - TagName: Workflows
 Name: Workflows
 - ETag: </Workflows>
 - Tag: </Workflows>
 - TagName: Workflows
 Name: Workflows
 - ETag: </ActiveWorkflowsData>
 - Tag: </ActiveWorkflowsData>
 - TagName: ActiveWorkflowsData
 Name: ActiveWorkflowsData
 - Element: XmlElement:<DefaultWorkflows>
 - STag: <DefaultWorkflows>
 - Tag: <DefaultWorkflows>
 - TagName: DefaultWorkflows
 Name: DefaultWorkflows
 - ETag: </DefaultWorkflows>
 - Tag: </DefaultWorkflows>
 - TagName: DefaultWorkflows
 Name: DefaultWorkflows
 - ETag: </WorkflowData>
 - Tag: </WorkflowData>
 - TagName: WorkflowData
 Name: WorkflowData
 - ETag: </GetWorkflowDataForItemResult>
 - Tag: </GetWorkflowDataForItemResult>
 - TagName: GetWorkflowDataForItemResult
 Name: GetWorkflowDataForItemResult
 - ETag: </GetWorkflowDataForItemResponse>
 - Tag: </GetWorkflowDataForItemResponse>
 - TagName: GetWorkflowDataForItemResponse
 Name: GetWorkflowDataForItemResponse
 - ETag: </soap:Body>
 - Tag: </soap:Body>
 - TagName: soap:Body
 NamespacePrefix: soap
 Name: Body
 - ETag: </soap:Envelope>
 - Tag: </soap:Envelope>
 - TagName: soap:Envelope
 NamespacePrefix: soap
 Name: Envelope

That’s it for the Client to Server interaction when downloading a document from a Document Library. Some additional resources are available below:

http://msdn.microsoft.com/en-us/library/cc339482(v=office.12).aspx

http://msdn.microsoft.com/en-us/library/ms442469.aspx

http://msdn.microsoft.com/en-us/library/websvccellstorage.aspx

http://msdn.microsoft.com/en-us/library/webs(v=office.12).aspx

http://msdn.microsoft.com/en-us/library/websvcwebs.webs.weburlfrompageurl.aspx

http://msdn.microsoft.com/en-us/library/workflow.workflow.getworkflowdataforitem(v=office.12).aspx

http://msdn.microsoft.com/en-us/library/ms457496.aspx

http://msdn.microsoft.com/en-us/library/ms460198.aspx

Thanks,

Russ Maxwell, MSFT

Guide to walking a SharePoint 2010 Search Query behind the scenes

2012-02-15T17:17:00Z

I know troubleshooting Search Query issues in SharePoint 2010 can be a difficult tasks. Usually, the problem I’m asked to tackle is why are my queries slow? I wanted to provide some additional detail and a behind the scenes look at what a Search Query looks like. I recommend reviewing my original Search Query blog if you need to brush up on the related components and how they work together.

http://blogs.msdn.com/b/russmax/archive/2010/04/23/search-2010-architecture-and-scale-part-2-query.aspx

Note: This scenario is strictly SharePoint 2010 without Fast

I figured the best way to diagram what happens behind the scenes is to use the Query Flow steps from my original blog and break them apart with a series of logging techniques. For Example (Network traces and ULS Logs)

When troubleshooting issues with Search Queries, It’s extremely important to identify how these moving parts operate together within the ULS logs and Network traces. Also, within the ULS logs we do a very good job of logging the execution times of each particular function “monitored scope”. The intent of this email is to provide you with a road map of how to walk a query behind the scenes with ULS logs and Network traces..

The main steps during Query Flow are the following:

1. A search is performed by a user

2. The WFE serving the call uses the associated search service application proxy to connect to a server running the Query and Site Settings Service also known as the Query Processor. It uses WCF for this communication.

3. The QP will connect to the following components to gather results merges\security trims and return results back to WFE:

Query Component - holds entire index or partition of an index

Property Store DB – holds metadata\properties of indexed content

Search Admin DB – holds Security Descriptors\Configuration data

4. WFE displays search results to the user

Below is a walkthrough of what a Query transaction looks like behind the scenes broken apart by the main steps above.

My current environment is setup like the following:

WFE – Hosting the public facing sites for end users
APP - Hosts both the Query Component and the Query Processor

Step 1 – A search is performed by a user

The following query was for “Dirk”

The following Request is submitted to the Web Front End the user is currently connected to:

Http: Request, GET /_layouts/OSSSearchResults.aspx, Query:k=Dirk&cs=This%20Site&u=http%3A%2F%2Fwfe, Using NTLM Authorization

Capturing this request with a network trace looks like:

Step 2 - The WFE serving the call uses the associated search service application proxy to connect to a server running the Query and Site Settings Service also known as the Query Processor. It uses WCF for this communication.

Behind the scenes, a query is submitted against the designated server running the Query Processor via a web service call to that servers SearchService.svc web service. Where is that located within IIS?

ULS Logs set to Verbose on the WFE

01/27/2012 11:30:45.18 w3wp.exe (0x1140) 0x120C SharePoint Foundation Logging Correlation Data xmnv Medium Name=Request (GET:http://wfe:80/_layouts/OSSSearchResults.aspx?k=Dirk&cs=This%20Site&u=http%3A%2F%2Fwfe) 9d7bacd8-0e70-4ffe-a293-cb2c9c41f205

Call to the Query Processor via the SearchService.svc web service to execute the request:

01/27/2012 11:31:17.12 w3wp.exe (0x1140) 0x120C SharePoint Foundation Monitoring nasq Verbose Entering monitored scope (SharePointSearchRuntime::GetQueryResult) 9d7bacd8-0e70-4ffe-a293-cb2c9c41f205

01/27/2012 11:31:17.15 w3wp.exe (0x1140) 0x120C SharePoint Foundation Monitoring nasq Verbose Entering monitored scope (Execute) 9d7bacd8-0e70-4ffe-a293-cb2c9c41f205

01/27/2012 11:31:17.15 w3wp.exe (0x1140) 0x120C SharePoint Foundation Topology e5mc Medium WcfSendRequest: RemoteAddress: 'http://app:32843/6c270a2117544235b521a8e5a864c5f6/SearchService.svc' Channel: 'Microsoft.Office.Server.Search.Administration.ISearchServiceApplication' Action: 'http://tempuri.org/ISearchQueryServiceApplication/Execute' MessageId: 'urn:uuid:5c0fbea8-1ca8-4146-8150-c6d8b735adae' 9d7bacd8-0e70-4ffe-a293-cb2c9c41f205

01/27/2012 11:31:17.15 w3wp.exe (0x1140) 0x120C SharePoint Foundation Monitoring nasq Verbose Entering monitored scope (ExecuteWcfOperation:http://tempuri.org/ISearchQueryServiceApplication/Execute) 9d7bacd8-0e70-4ffe-a293-cb2c9c41f205

Network Monitor

We send multiple Post request to the QP along with HTTP Payload. The Post request with HTTP Payload that matters is the last one sent in the network conversation. For example, the HTTP Payload contains the following:

117 12:31:05 PM 1/27/2012 4.0724607 w3wp.exe 10.1.0.12 10.1.0.11 HTTP HTTP:Request, POST /6c270a2117544235b521a8e5a864c5f6/SearchService.svc {HTTP:11, TCP:10, IPv4:9}

119 12:31:05 PM 1/27/2012 4.0746818 w3wp.exe 10.1.0.12 10.1.0.11 HTTP HTTP:HTTP Payload, URL: /6c270a2117544235b521a8e5a864c5f6/SearchService.svc {HTTP:11, TCP:10, IPv4:9}

Frame 119 is important as the payload information contains the method “Execute” and the QueryText “dirk”:

http://tempuri.org/ISearchQueryServiceApplication/ExecuteD.º?$íe¡ïH¡´?¸Ìúb?D,D*«..@.ServiceContext.6http://schemas.microsoft.com/sharepoint/servicecontext..i)

http://schemas.datacontract.org/2004/07/Microsoft.Office.Server.Search.Query._.PartitionId?$00000000-0000-0000-0000-000000000000_.PersonalizationData..nil?._.QueryText?.dirk_.RankingModelId._.ResultTypes?.RelevantResults_.ResultsProvider?.SharepointSearch_.RowLimit?2_.RowsPerPage?._.Sid._.SiteId?$195a8767-f198-4491-a994-353973710934_

Step 3 - The QP will connect to the following components to gather results merges\security trims and return results back to WFE

Query Component - holds entire index or partition of an index

Property Store DB – holds metadata\properties of indexed content

Search Admin DB – holds Security Descriptors\Configuration data

ULS Logs set to Verbose on the Server hosting the Query Processor

Here is the inbound request from the WFE.

01/27/2012 09:31:16.93 w3wp.exe (0x1544) 0x1644 SharePoint Foundation Topology e5mb Medium WcfReceiveRequest: LocalAddress: 'http://app.contoso.local:32843/6c270a2117544235b521a8e5a864c5f6/SearchService.svc' Channel: 'System.ServiceModel.Channels.ServiceChannel' Action: 'http://tempuri.org/ISearchQueryServiceApplication/Execute' MessageId: 'urn:uuid:5c0fbea8-1ca8-4146-8150-c6d8b735adae' 9d7bacd8-0e70-4ffe-a293-cb2c9c41f205

01/27/2012 09:31:16.93 w3wp.exe (0x1544) 0x1644 SharePoint Foundation Monitoring nasq Medium Entering monitored scope (ExecuteWcfServerOperation) 9d7bacd8-0e70-4ffe-a293-cb2c9c41f205

When the Query Processor starts on the request, it will get a new correlation ID:

01/27/2012 09:31:17.13 w3wp.exe (0x1544) 0x1644 SharePoint Server Search Query Processor e2ni Verbose Calling CQPCache::GetQPCache. 599cd6ce-7c0c-44e8-8983-0be66f7176ad

Here is where Query Processor will start fetching results from the Query Component\s holding the index to retrieve results as well as the Search Property Store Database on the SQL server:

01/27/2012 09:31:18.02 w3wp.exe (0x1544) 0x1644 SharePoint Server Search Query Processor e2pp Verbose Entering Split Command Tree parser. [splitter.cxx:184] d:\office\source\search\native\ytrip\tripoli\icommand\splitter.cxx 599cd6ce-7c0c-44e8-8983-0be66f7176ad

01/27/2012 09:31:18.43 w3wp.exe (0x1544) 0x1644 SharePoint Server Search Query Processor e2ps Verbose Tree only had both FT and SQL constraints... split apart. [splitter.cxx:230] d:\office\source\search\native\ytrip\tripoli\icommand\splitter.cxx 599cd6ce-7c0c-44e8-8983-0be66f7176ad

QP Fetching results from the Index:

01/27/2012 09:31:18.43 w3wp.exe (0x1544) 0x1644 SharePoint Server Search Query Processor e2jx Verbose Executing SubQuery FT Part: op: (116) DBOP_and wKind: (3) DBVALUEKIND_I4: 0 first child: op: (145) DBOP_content_freetext wKind: (259) DBVALUEKIND_CONTENT: 'Dirk ', method 0, weight 100000000, lcid 0x409 first child: op: (5) DBOP_column_name wKind: (258) DBVALUEKIND_ID: 012357bd-1113-171d-1f25-292bb0b0b0b0 2147450879 next sibling: op: (68) DBOP_equal wKind: (3) DBVALUEKIND_I4: 0 first child: op: (5) DBOP_column_name wKind: (258) DBVALUEKIND_ID: 012357bd-1113-171d-1f25-292bb0b0b0b0 2147418095 next sibling: op: (0) DBOP_scalar_constant wKind: (12) DBVALUEKIND_VARIANT VT_LPWSTR 'http://wfe' 599cd6ce-7c0c-44e8-8983-0be66f7176ad

01/27/2012 09:31:18.66 w3wp.exe (0x1544) 0x1644 SharePoint Server Search Query Processor e2kp Verbose Got 14 candidates from FT. 599cd6ce-7c0c-44e8-8983-0be66f7176ad

01/27/2012 09:31:18.66 w3wp.exe (0x1544) 0x1644 SharePoint Server Search Query Processor e2ks Verbose Got some.. Now joining with SQL. 599cd6ce-7c0c-44e8-8983-0be66f7176ad

01/27/2012 09:31:18.66 w3wp.exe (0x1544) 0x1644 SharePoint Server Search Query Processor e2et Verbose Candidates Query to SQL: /* {599CD6CE-7C0C-44E8-8983-0BE66F7176AD} */ SET NOCOUNT ON;DECLARE @joinData VARBINARY(MAX) ; SET @joinData = ? ; DECLARE @joinRows INT ; SET @joinRows = DATALENGTH(@joinData) / 8 ;;WITH DocIds(DocId, Value) AS ( SELECT TOP(@joinRows) CAST(SUBSTRING(@joinData, ((ord.n-1)*8) + 1, 4) AS INT), CAST(SUBSTRING(@joinData, ((ord.n-1)*8) + 5, 4) AS INT) FROM dbo.MSSOrdinal AS ord WITH(NOLOCK) WHERE ord.n <= @joinRows ) SELECT A.DuplicateHashes, A.HasPluggableSecurityTrimming, A.DocId, A.Sdid FROM DocIds AS T INNER LOOP JOIN MssDocSdids AS A ON T.DocId = A.DocId ORDER BY T.Value DESC OPTION (MAXDOP 1) 599cd6ce-7c0c-44e8-8983-0be66f7176ad

01/27/2012 09:31:18.68 w3wp.exe (0x1544) 0x1644 SharePoint Server Search Query Processor e2l2 Verbose Got 14 candidates after join. 599cd6ce-7c0c-44e8-8983-0be66f7176ad

Now Security Trimming and Dup removal of the results:

01/27/2012 09:31:18.68 w3wp.exe (0x1544) 0x1644 SharePoint Server Search Query Processor e2p4 Verbose In CSecurityTrimmer::AccessCheck there were 1 unique sdids among the candidates. 599cd6ce-7c0c-44e8-8983-0be66f7176ad

01/27/2012 09:31:18.69 w3wp.exe (0x1544) 0x1644 SharePoint Server Search Query Processor e2ke Verbose Had 14 candidates before sec-trimming, have 14 candidates after. 'fMoreCandidates' = 0. 599cd6ce-7c0c-44e8-8983-0be66f7176ad

01/27/2012 09:31:18.69 w3wp.exe (0x1544) 0x1644 SharePoint Server Search Query Processor e2la Verbose Had 14 candidates before near dup removal, have 5 candidates after. 'fMoreCandidates' = 0. 599cd6ce-7c0c-44e8-8983-0be66f7176ad

QP Fetching results from SQL Server:

01/27/2012 09:31:18.69 w3wp.exe (0x1544) 0x1644 SharePoint Server Search Query Processor e2jx Verbose Executing SubQuery SQL Part: op: (68) DBOP_equal wKind: (3) DBVALUEKIND_I4: 0 first child: op: (5) DBOP_column_name wKind: (258) DBVALUEKIND_ID: 012357bd-1113-171d-1f25-292bb0b0b0b0 7 next sibling: op: (0) DBOP_scalar_constant wKind: (12) DBVALUEKIND_VARIANT VT_LPWSTR 'http://Dirk' FT Part: op: (68) DBOP_equal wKind: (3) DBVALUEKIND_I4: 0 first child: op: (5) DBOP_column_name wKind: (258) DBVALUEKIND_ID: 012357bd-1113-171d-1f25-292bb0b0b0b0 2147418095 next sibling: op: (0) DBOP_scalar_constant wKind: (12) DBVALUEKIND_VARIANT VT_LPWSTR 'http://wfe' 599cd6ce-7c0c-44e8-8983-0be66f7176ad

01/27/2012 09:31:18.69 w3wp.exe (0x1544) 0x1644 SharePoint Server Search Query Processor e2et Verbose Candidates Query to SQL: /* {599CD6CE-7C0C-44E8-8983-0BE66F7176AD} */ SET NOCOUNT ON; SELECT TOP 2797 A.DuplicateHashes, A.HasPluggableSecurityTrimming, A.DocId, A.Sdid FROM MssDocSdids AS A WITH (NOLOCK) LEFT OUTER JOIN dbo.MSSDocProps AS P7 WITH (NOLOCK) ON ( P7.DocId = A.DocId AND P7.Pid = 7 ) WHERE P7.llVal = ? 599cd6ce-7c0c-44e8-8983-0be66f7176ad

01/27/2012 09:31:18.71 w3wp.exe (0x1544) 0x1644 SharePoint Server Search Query Processor e2jd Verbose Got 0 candidates from this sub-query. 599cd6ce-7c0c-44e8-8983-0be66f7176ad

Sort and Rank the Search Results:

01/27/2012 09:31:18.71 w3wp.exe (0x1544) 0x1644 SharePoint Server Search Query Processor e2jk Verbose Performing 'Multi Sub-Query sort-by-rank' final candidates code-path. 599cd6ce-7c0c-44e8-8983-0be66f7176ad

01/27/2012 09:31:18.71 w3wp.exe (0x1544) 0x1644 SharePoint Server Search Query Processor e2jw Verbose Fetching Search-Results from property store. 599cd6ce-7c0c-44e8-8983-0be66f7176ad

Query has been Completed by the Query Processor and Query Timings are logged:

01/27/2012 09:31:18.74 w3wp.exe (0x1544) 0x1644 SharePoint Server Search Query Processor e2jt Medium Completed query execution with timings: total:718 dup:0 sec:16 join:0 ft:234 sql:31. Join Retry: 0. Security Trimming Retry: 0. Duplicate removal Retry: 0. 599cd6ce-7c0c-44e8-8983-0be66f7176ad

01/27/2012 09:31:18.76 w3wp.exe (0x1544) 0x1644 SharePoint Server Search Query Processor g2jn Verbose Verbose query timings: total: 766, other: 454, ft: 0, sql: 31, join: 234, dup: 0, sec: 16, final sort: 0, best bets: 0, results retr: 0, high conf: 0, populate: 31. 599cd6ce-7c0c-44e8-8983-0be66f7176ad

QP Execute is completed

01/27/2012 09:31:18.99 w3wp.exe (0x1544) 0x1644 SharePoint Foundation Monitoring b4ly Medium Leaving Monitored Scope (ExecuteWcfServerOperation). Execution Time=2069.4805 599cd6ce-7c0c-44e8-8983-0be66f7176ad

The original inbound request is completed (Notice the Correlation ID matches the original execute call to the Web Service – original call above in red)

Important: You might notice that the Query Processor decides to execute against SQL first and then the Full-Text index. The decision making on why this happens is outside the scope of this blog but it’s dependent on the type of Search Query performed.

Step 4 - WFE displays search results to the user

ULS Logs set to Verbose on the WFE that originally serviced the request from Step 1

When the Query Processor returns the results back to the WFE (Notice this is the completion of the monitored scope in green) The beginning of this request is at the top in green.

01/27/2012 11:31:19.52 w3wp.exe (0x1140) 0x120C SharePoint Foundation Monitoring b4ly Verbose Leaving Monitored Scope (ExecuteWcfOperation:http://tempuri.org/ISearchQueryServiceApplication/Execute). Execution Time=2400.0826 9d7bacd8-0e70-4ffe-a293-cb2c9c41f205

01/27/2012 11:31:19.81 w3wp.exe (0x1140) 0x120C SharePoint Server Search Query dk91 High SearchServiceApplicationProxy::Execute--Id: Elapsed Time: 2664 QP Time: 1792 Sql Time: 31 9d7bacd8-0e70-4ffe-a293-cb2c9c41f205

01/27/2012 11:31:19.81 w3wp.exe (0x1140) 0x120C SharePoint Foundation Monitoring b4ly Verbose Leaving Monitored Scope (Execute). Execution Time=2703.0805 9d7bacd8-0e70-4ffe-a293-cb2c9c41f205

01/27/2012 11:31:19.95 w3wp.exe (0x1140) 0x120C SharePoint Foundation Monitoring b4ly Verbose Leaving Monitored Scope (SharePointSearchRuntime::GetQueryResult). Execution Time=2856.5793 9d7bacd8-0e70-4ffe-a293-cb2c9c41f205

01/27/2012 11:31:20.09 w3wp.exe (0x1140) 0x120C SharePoint Foundation Monitoring b4ly Verbose Leaving Monitored Scope (SharePointSearchRuntime::SendRequest). Execution Time=3025.3754 9d7bacd8-0e70-4ffe-a293-cb2c9c41f205

01/27/2012 11:31:20.09 w3wp.exe (0x1140) 0x120C SharePoint Server Search Query dln3 Verbose GetResults on location LocalSearchIndex for Dirk completed with 5 results 9d7bacd8-0e70-4ffe-a293-cb2c9c41f205

The WFE prepares the results for the User

The WFE takes the above results and builds up the ossearchresults.aspx page by using the following monitored scopes:

RefinementManager.GetRefinement
OnPreRender
CreateChildControls
AddAssemblyReference
DataBinding DataFormWebPart
GetXPathNavigator
UserPreference.GetUserPreference
UserPreference.GetFromCache
Render Ribbon
SearchCommon::GetCurrentSiteCollectionProperty
RenderWebPart
PostRequestExecuteHandler
EndRequestHandler SPRequest Disposal

Finally when the page is rendered we leave the original Monitored Scope

01/27/2012 11:31:33.40 w3wp.exe (0x1140) 0x120C SharePoint Foundation Monitoring b4ly Medium Leaving Monitored Scope (Request (GET:http://wfe:80/_layouts/OSSSearchResults.aspx?k=Dirk&cs=This%20Site&u=http%3A%2F%2Fwfe)). Execution Time=48385.3272 9d7bacd8-0e70-4ffe-a293-cb2c9c41f205

Q&A Portion

Question: So how were you able to even identify the start of the Query conversation within Step 2 on the Web Front-End?

Answer: First, download ULSViewer here: http://archive.msdn.microsoft.com/ULSViewer

Next, once you open the associated ULS log on the Web Front-End, you can hit (CTRL+M) to bring up the filter window. Then do a query like the following:

Message Contains dirk. (In my case, I submitted a query dirk.)

Question: How did you find the other side of the conversation on the Query Processor under Step 3?

Answer: I used ULS Viewer and opened the associated ULS log on the Query Processor. I then filtered (CTRL+M) for the following:

Correlation ID Contains 9d7bacd8-0e70-4ffe-a293-cb2c9c41f205

Question: How did you find the associated Query Processor entries assuming more than one query is firing against the QP?

Answer: I used ULS Viewer and did a CTRL+F and queried for dirk. I found it immediately from this entry:

01/27/2012 09:31:18.43 w3wp.exe (0x1544) 0x1644 SharePoint Server Search Query Processor e2jx Verbose Executing SubQuery FT Part: op: (116) DBOP_and wKind: (3) DBVALUEKIND_I4: 0 first child: op: (145) DBOP_content_freetext

wKind: (259) DBVALUEKIND_CONTENT: 'Dirk ', method 0, weight 100000000, lcid 0x409 first child: op: (5) DBOP_column_name wKind: (258) DBVALUEKIND_ID: 012357bd-1113-171d-1f25-292bb0b0b0b0 2147450879 next sibling: op: (68) DBOP_equal

wKind: (3) DBVALUEKIND_I4: 0 first child: op: (5) DBOP_column_name wKind: (258) DBVALUEKIND_ID: 012357bd-1113-171d-1f25-292bb0b0b0b0 2147418095 next sibling: op: (0) DBOP_scalar_constant wKind: (12) DBVALUEKIND_VARIANT VT_LPWSTR 'http://wfe' 599cd6ce-7c0c-44e8-8983-0be66f7176ad

With this log entry, you can see the category is Query Processor and the query term is defined. I know from the entry this is request against the Full-Text Index. Since this is my matching entry, I can grab the correlation ID off the end.

Now that I have the correlation ID, I can filter the ULS log with ULS viewer to see the entire conversation:

Correlation ID Contains 599cd6ce-7c0c-44e8-8983-0be66f7176ad

Thanks!

Russ Maxwell, MSFT

SharePoint PowerShell Script Series Part 5–Exporting the crawl log to a CSV file

2012-01-29T01:57:01Z

I recently had a request to provide a SharePoint Administrator the ability to export out a crawl log to a CSV file using PowerShell. Luckily, I found Vijay’s (Thanks Vijay!) post which saved me a ton of time:

http://blogs.msdn.com/b/spses/archive/2011/06/22/exporting-sharepoint-2010-search-crawl-logs.aspx

I wanted to add some more functionality to this process of exporting out to a CSV. I wrote the following PowerShell script which gives the Admin more options like choosing which Search Service Application and filtering by URL or by Content Source.

Updated on 2/18/2012 – Thanks to my colleague Heiko at Microsoft. He pointed out the lack of functionality when multiple crawl stores exists within the Search Service Application. I updated the script to now work with Search Service Applications containing multiple crawl stores.

This script does the following:

1.    Let you decide which Search Service Application to use
2.    Let you decide if you want to filter against URL or by Content Source
3.    If you choose URL, you have an option of only exporting Errors or All Events (Skip to Step 5)
4.    If you choose Content Source, you decide which content source to filter against
        4.a. After Choosing Content Source, you have an option of only exporting Errors or All Events

5. Finally, you can choose the path and file name to use and the file will export the result set as a CSV which can be opened in Excel

Instructions for running the script:

1.    Copy the below script and save it in notepad
2.    Save it with a anyfilename.ps1 extension
3.    To run, copy the file to a SharePoint Server
4.    Select Start\Microsoft SharePoint 2010 Products\SharePoint 2010 Management Shell
5.    Browse to directory holding the copied script file
6.    Run the script: .\anyfilename.ps1 (assuming anyfilename is the name of the file)

<# ==============================================================
//
// Microsoft provides programming examples for illustration only,
// without warranty either expressed or implied, including, but not
// limited to, the implied warranties of merchantability and/or
// fitness for a particular purpose.
//
// This sample assumes that you are familiar with the programming
// language being demonstrated and the tools used to create and debug
// procedures. Microsoft support professionals can help explain the
// functionality of a particular procedure, but they will not modify
// these examples to provide added functionality or construct
// procedures to meet your specific needs. If you have limited
// programming experience, you may want to contact a Microsoft
// Certified Partner or the Microsoft fee-based consulting line at
// (800) 936-5200.
//
// For more information about Microsoft Certified Partners, please
// visit the following Microsoft Web site:
// https://partner.microsoft.com/global/30000104
//
// Author: Russ Maxwell (russmax@microsoft.com)
//
// ---------------------------------------------------------- #>

[Void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint")

Start-SPAssignment -Global

#############################
#Function to Export the Data#
#############################
function exportThis
{
    $output = Read-Host "Enter a location for the output file (For Example: c:\logs\)"
    $filename = Read-Host "Enter a filename"
    $stores = $ssa.CrawlStores
    $storectr = $stores.count
    $name = $output + "\" + $filename + ".csv"

    if($storectr -eq '1')
    {
        $logViewer = New-Object Microsoft.Office.Server.Search.Administration.Logviewer $ssa
        $i = 0
        $urlOutput = $logViewer.GetCurrentCrawlLogData($crawlLogFilters, ([ref] $i))
        Write-Host "# of Crawl Entries Produced" $urlOutput.Rows.Count
        $urlOutput | Export-Csv $name -NoTypeInformation
        Write-Host "Your results were exported to: " $name
    }

    elseif($storectr -gt '1')
    {
        $f = 1
        foreach($store in $stores)
        {
          Write-Host "In the " $f " iteration of store object"
          $logViewer = New-Object Microsoft.Office.Server.Search.Administration.Logviewer $store
          $i = 0
          $urlOutput = $logViewer.GetCurrentCrawlLogData($crawlLogFilters, ([ref] $i))
          $ctr = $urlOutput.Rows.Count
          $officialCTR += $ctr

            if($f -eq '1')
              {
                $finalDT = New-Object System.Data.DataTable
                $finalDT = $urlOutput.Copy()
              }
              else
              {
               $finalDT.Merge($urlOutput)
              }

          $f++

          }
    }
          $finalDT | Export-Csv $name -NoTypeInformation
          Write-Host "# of Crawl Entries Produced" $officialCTR
}

#####################################
#Choose a Search Service Application#
#####################################
$ssa = Get-SPEnterpriseSearchServiceApplication
$ssaName = $ssa | ForEach-Object {$_.Name}
Write-Host "Choose a Search Service Application to review crawl logs"
Write-Host
$num = 1

Foreach($sa in $ssa)
{ Write-Host $num $sa.Name
$num++
}

Write-Host
$result = Read-Host "Enter the number next to the desired Search Service Application and press enter"

$num = 1
Foreach($i in $ssa)
{
if($num -eq $result)
{
$ssa = $i
}
$num++
}
Write-Host

###############################################
#Create a Logviewer and Crawl Log FilterObject#
###############################################
$crawlLogFilters = New-Object Microsoft.Office.Server.Search.Administration.CrawlLogFilters

######################
#Let the Admin choose#
######################
Write-Host "How would you like to filter the crawl log?"
Write-Host "1 Filter Based on a URL"
Write-Host "2 Filter Based on Content Source"
Write-Host "3 Export without a Filter"
Write-Host
$choice = Read-Host "Enter 1, 2, or 3 and press enter"
Write-Host
Write-Host
Write-Host "1 Export only errors"
Write-Host "2 Export All (Success, Warning, and Errors"
$type = Read-Host "Enter 1 or 2 and press enter"
Write-Host

if($choice -eq '1')
{
                $url = Read-Host "Enter the URL to filter on"

                ###################################
                #Create Property and add to filter#
                ###################################
                $totalentryProp = New-Object Microsoft.Office.Server.Search.Administration.CrawlLogFilterProperty
                $totalentryProp = [Microsoft.Office.Server.Search.Administration.CrawlLogFilterProperty]::TotalEntries
                $urlProp = New-Object Microsoft.Office.Server.Search.Administration.CrawlLogFilterProperty
                $urlProp = [Microsoft.Office.Server.Search.Administration.CrawlLogFilterProperty]::Url
                $stringOp = New-Object Microsoft.Office.Server.Search.Administration.StringFilterOperator
                $stringOp = [Microsoft.Office.Server.Search.Administration.StringFilterOperator]::Contains
                $crawlLogFilters.AddFilter($urlProp, $stringOp,$url)
                $crawlLogFilters.AddFilter($totalentryProp, "1,000,000")

                if($type -eq '1')
                {
                                $typeEnum = New-Object Microsoft.Office.Server.Search.Administration.MessageType
                                $typeEnum = [Microsoft.Office.Server.Search.Administration.MessageType]::Error
                                $crawlLogFilters.AddFilter($typeEnum)
                }

                #Calling exportThisfunction
                exportThis
}

elseif($choice -eq '2')
{

                #########################
                #Choose a content source#
                #########################
                $content = New-Object Microsoft.Office.Server.Search.Administration.Content($ssa)
                $contentsources = $content.ContentSources

                Write-Host "Choose a Content Source to filter on"
                Write-Host
                $num = 1

                Foreach($c in $contentsources)
                {
                                Write-Host $num": " $c.Name
                                $num++
                }

$result = Read-Host "Enter the associated # press enter"

                $num = 1
                Foreach($c in $contentsources)
                {
               if($num -eq $result)
               {
                  $contentSource = $c
               }
               $num++
                }
                Write-Host "You chose" $contentSource.Name
                $id = $contentSource.Id


                ###################################
                #Create Property and add to filter#
                ###################################
                $totalentryProp = New-Object Microsoft.Office.Server.Search.Administration.CrawlLogFilterProperty
                $totalentryProp = [Microsoft.Office.Server.Search.Administration.CrawlLogFilterProperty]::TotalEntries
                $csProp = New-Object Microsoft.Office.Server.Search.Administration.CrawlLogFilterProperty
                $csProp = [Microsoft.Office.Server.Search.Administration.CrawlLogFilterProperty]::ContentSourceId
                $crawlLogFilters.AddFilter($csProp, $id)
                $crawlLogFilters.AddFilter($totalentryProp, "1,000,000")

                if($type -eq '1')
                {
                                $typeEnum = New-Object Microsoft.Office.Server.Search.Administration.MessageType
                                $typeEnum = [Microsoft.Office.Server.Search.Administration.MessageType]::Error
                                $crawlLogFilters.AddFilter($typeEnum)
                }

                #Calling exportThisfunction#
                exportThis

}

elseif($choice -eq '3')
{
                $catProp = New-Object Microsoft.Office.Server.Search.Administration.CatalogType
                $catProp = [Microsoft.Office.Server.Search.Administration.CatalogType]::PortalContent
                $crawlLogFilters.AddFilter($catProp)
                $catProp2 = New-Object Microsoft.Office.Server.Search.Administration.CatalogType
                $catProp2 = [Microsoft.Office.Server.Search.Administration.CatalogType]::ProfileContent
                $crawlLogFilters.AddFilter($catProp2)
                $totalentryProp = New-Object Microsoft.Office.Server.Search.Administration.CrawlLogFilterProperty
                $totalentryProp = [Microsoft.Office.Server.Search.Administration.CrawlLogFilterProperty]::TotalEntries
                $crawlLogFilters.AddFilter($totalentryProp, "1,000,000")

                if($type -eq '1')
                {
                                $typeEnum = New-Object Microsoft.Office.Server.Search.Administration.MessageType
                                $typeEnum = [Microsoft.Office.Server.Search.Administration.MessageType]::Error
                                $crawlLogFilters.AddFilter($typeEnum)
                }

                #Calling exportThisfunction
                exportThis
}

Stop-SPAssignment –Global

Thanks!

Russ Maxwell, MSFT