Microsoft Enterprise Search Blog

Crawling case sensitive repositories using SharePoint Server 2010

enterprisesearch — Fri, 09 Jul 2010 22:32:00 GMT

Introduction

There are many repositories which are case sensitive. To elaborate with an example, in case sensitive repositories the links:

http://myhost/CaseSensitivePage.htm

and

http://myhost/casesensitivepage.htm

represent different pages. The crawlers while indexing such repositories have to preserve the case of the discovered links to keep them valid. Examples of such repositories are:

- web sites deployed on Apache Server

- Linux file shares

- Business Data Catalogs

- Etc.

By default, SharePoint Search crawler normalizes all the links that it discovers and converts them to all lower case. This normalization does not allow crawling of the case sensitive repositories. SharePoint Server 2007 SP1 had limited support for crawling case sensitive repositories that allowed administrators to set a registry key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office Server\12.0\Search\Applications\<GUID>\Gather\Portal_Content\CaseSensitiveURLs

When set, this registry key will ensure that all the crawling operations preserve case. More details about this provision can be found in this KB article.

SharePoint Server 2010 solution

SharePoint Server 2010 extends this functionality by giving more flexibility to the admins so that they can explicitly specify patterns of repositories/hosts/links that they want to crawl while preserving case of links. This is achieved by creating case sensitive crawl rules. The global case preservation flag is set-able via OM and PowerShell (discussed at the end of post).

Case sensitive crawl rules

SharePoint Server 2010 extends crawl rules to add two major useful features: Regular Expressions and Case Preservation. Regular Expressions in crawl rules are discussed in this blog post, so we will not talk in detail about it here and focus instead on case preservation support. The crawl rules can be made case sensitive and the links that match with that particular rule are then case preserved.

Creating case sensitive crawl rule via UI

A case sensitive crawl rule can be created through the crawl rule UI and it’s new “Match Case” checkbox. The screenshot below shows the checkbox for enabling case preservation in crawling.

Creating case sensitive crawl rule via PowerShell

Case sensitive crawl rules can also be created using PowerShell. The script below creates a case sensitive crawl rule for a host called “mycasesensitivehost”:

# Get the search application object

$app = Get-SPEnterprisesearchServiceApplication "<SSA Name>”

# Create an inclusion crawl rule

$rule = New-SPEnterpriseSearchCrawlRule -SearchApplication $app -Path “http://mycasesensitivehost/*” -Type 0

# Set CaseSensitiveURL to true

$rule.CaseSensitiveURL = $true

#Update the rule to reflect changes

$rule.Update()

How does it work?

If an admin wants a particular repository (for example: http://mycasesensitivehost/) to be crawled with the links’ case preserved, the admin needs to perform following three steps:

1. Create an inclusion crawl rule that matches links belonging to the repository: http://mycasesensitivehost/*

2. Check the “Match Case” checkbox.

3. Create content source for crawling with start address for the repository (eg: http://mycasesensitivehost/StartPage.htm” and start a FULL crawl! The crawler has to go through a full crawl on the first run n order for the case preservation setting to take effect.

The crawler by default normalizes the links and converts them to lower case and any links that do not match will be normalized and converted to lower case.

How does it appear in search result?

The crawler, if configured to Match Case, will preserve cases for links in search results when case differences mean different documents – as in the example below.

How does it appear in crawl logs?

Crawl logs show the links with preserved case. Please see the image below:

Notice that the links coming from: “mycasesensitivehost” are case preserved and ones coming out of “mynormalhost” are all lower case.

Setting global case preservation via OM/PowerShell

The Search Application Object exposes a property: “CaseSensitiveCrawling” that is get- and set-able via SearchApplication.GetProperty and SearchApplication.SetProperty methods. Below is an example of manipulating this property via PowerShell.

# Get the search application object

PS D:\> $searchApp = Get-SPEnterpriseSearchServiceApplication -local

# Check the property value; which is False by default.

PS D:\> $searchApp.GetProperty("CaseSensitiveCrawling")

False

# Now set the property to true

PS D:\> $searchApp.SetProperty("CaseSensitiveCrawling", 1)

# Check if property is set.

PS D:\> $searchApp.GetProperty("CaseSensitiveCrawling")

True

When “CaseSensitiveCrawling” property is set, the crawler will preserve case of all links crawled by the given search application.

I hope this post sheds light on how to configure case sensitive crawls in SharePoint 2010. I welcome any questions and comments.

Syed Anas Hashmi

Software Design Engineer in Test

Microsoft Corp.

SharePoint Search 2010 in a Small Scale Farm - Hardware

enterprisesearch — Wed, 16 Jun 2010 01:24:00 GMT

Hello, my name is Hernando Silva and, as some of you might know from Dan's previous posts , for the last year I participated in the planning, deployment and management of the SharePoint 2010 Search service at a small scale.

In this post I will begin to introduce you to our Divisional (a.k.a. Departmental) farm for the Office organization. This deployment was intentionally planned to be oversized to make sure that it would scale to accommodate pre-released builds of SharePoint 2010 and other products (see the capacity planning document for the definition of a Divisional Portal and, also, take a look at this case study for more details about this specific deployment). In working with Dan in this series of posts, I hope to provide additional knowledge based on what we learned in some of the smaller scale deployments.

We wanted to use this farm to ensure that we would satisfy our goals for query latency and crawl freshness, on a deployment of this scale.

We did interesting experiments in this deployment, like using Solid State Disks for our SQL server, use site data to redirect the crawl to the crawl target server, use resource governor in SQL to control the behavior of the crawl, etc. I will try to explain all those experiments in further posts so you can learn about them and use them according to your needs.

Hardware & Topology

The Office organization has about 7.3 M items stored in the farm, distributed over 25 Content DBs, which require about 1.9 TB in our Content SQL server.

The SharePoint 2010 Search service disk space usage was as follows:

SQL Server:

· Admin DB: 1.8 GB

· Crawl DB: 167 GB

· Property Store DB: 34 GB

Query Server:

· About 25 GB of per Query Server for the index.

From the Search point of view the hardware was designed to help us satisfy 2 major requests, sub-second query latencies and crawl freshness of about 5 minutes.

The roles for the servers in our Divisional farm are:

· 1 Web Front-Ends.

· 2 Web Front-Ends combined with a Query Server & the Search Query and Site Settings Service

· 1 Web Front dedicated as a Search crawl target (this server does not participate in NLB)

· 3 App servers, which are used for non-Search activities

· 1 Search crawler

· 3 SQL servers

o Content Databases

o Usage and Analytics Databases

o Search Databases.

Graphically, the topology of the farm looks like:

Below, is the detailed description of the servers that are being used by the SharePoint 2010 Search service.

Web Servers Specs

There are four Web servers in the farm. 3 are used to serve content (2 of which are also used as the Search Query/Search Query and Site Settings Servers). The other server is used as the search crawl target (to reduce the load on our content servers while search is crawling)

Server	WFE/Query Server/QP	Crawl Target
Processor(s)	2 quad core @2.33 GHz	2 quad core @2.33 GHz
RAM	32 GB	16 GB
Operating system	Windows Server® 2008, 64 bit	Windows Server 2008, 64 bit
Size of the SharePoint drive	6x146GB 15K SAS (3 RAID 1 Disks) Disk 1: OS and Product Disk 2: Swap, BLOB Cache and Index Disk 3: Logs and Temp directory	3x146GB 15K SAS (3 RAID 1 Disks) Disk 1: OS and Product Disk 2: Swap and BLOB Cache Disk 3: Logs and Temp directory
Number of NICs	2	2
NIC Speed	1 Gigabit	1 Gigabit
Authentication	NTLM	NTLM
Software version	SharePoint Server 2010 (pre-release version)	SharePoint Server 2010 (pre-release version)
Services running locally	Query Server/Search Query and Site Settings	No services

Search Crawler Specs

There is one SharePoint 2010 Search crawler server in the farm.

Server	Search Crawler
Processor(s)	2 quad core @2.5GHz Xeon
RAM	16 GB
Operating system	Windows Server 2008 R2, 64 bit
Size of the SharePoint drive	4x136GB 15K SAS (2 RAID 1 Disks) 2x418GB 15K, SAS (RAID 1) * Disk 1: OS and Product Disk 2: Swap and BLOB Cache Disk 3: Logs and Temp directory
Number of NICs	2
NIC Speed	1 Gigabit
Authentication	NTLM
Software version	SharePoint Server 2010 (pre-release version)
Services running locally	Crawl Server

* These drives were big ,not because of a requirement from the product, but because they were readily available at the time when the hardware was being configured

Database Servers Specs

This is the database server that is dedicated to SharePoint 2010 Search.

Server	SQL Server - Search DBs
Processor(s)	2 quad core @3.2 GHz
RAM	32 GB
OS	Windows Server 2008 R2, 64-bit
Storage and geometry	2x136GB 15K SAS (RAID 0) 6x60GB Solid State Disks, SATA (RAID 5) Disk 1: OS. SQL Server y Swap Disk 2: Search and Temp DBs and log files
Number of NICs	2
NIC Speed	1 Gigabit
Authentication	NTLM
Software version	SQL Server 2008 R2 (Pre-Release)

At this point, many of you might be wondering why did we decided to go with this topology. No worries… in the next posting I will drill down into the topology components and the reasoning behind those choices, so … stay tuned!

Hernando Silva

Software Engineer in Test

Microsoft Corp

SharePoint 2010 Search 'Dogfood' Part 3 - Query Performance Optimization

enterprisesearch — Wed, 09 Jun 2010 17:00:00 GMT

Hello again and welcome to the SharePoint Server 2010 Search series of blog postings dedicated to detailing the lessons I have learned in the process of hosting our dogfood system. In this posting I will be talk about the high level concepts utilized to optimize query performance. For reference and a complete description of the hardware please refer to the SearchBeta Hardware post to see the hardware specs and topology layout.

Index partitions, query components & mirrors

The first item that one needs to understand when building out a SharePoint 2010 Search service is how the index is split into sub pieces to provide redundancy and sub second query latencies. The following are the building blocks used:

Index Partitions - An index partition is a logical fractional piece of the entire index. Documents are added in a round-robin fashion across components serving each partition to ensure that each partition is equivalent in size. Each user query must span all partitions that make up the full index.
Query Components - The modules that satisfy a query. Each partition must have a component to provide end-user queries over it. The SharePoint 2010 Search system does not allow a partition to exist without having an associated component.
Mirrors - A redundant copy of a component; there can be more than one mirror for any partition. Mirrors can be active (FailoverOnly == false) or passive. Active mirrors address throughput (if your bottleneck is CPU in the query component) as end user queries will be sent to one of the active mirrors using a round robin algorithm. Passive mirrors address availability, as a passive mirror will serve queries if no other active components exists for the given partition. (See below for more details).

So the main way to improve query latency is to reduce the size of each partition by adding more of them. With more partitions you add more parallelization for the query and reduce the size of the index that each query component must process.

If you need to improve query throughput you do so by simply adding more mirrors. Thus increasing the number of components that can concurrently satisfy incoming queries.

Determining the need for more partitions

The above is a simplistic look at how to improve latency and throughput. There are several additional details that you should use to determine the need to add more components into the system. The capacity planning document identifies three main guidelines to direct the number of partitions you should have:

33% of the active partitions contained on a given machine should fit into the available memory on the machine
Each partition is recommended to be smaller than 10 million documents
Active components will consume 2 cores; one for servicing queries and one for merging the index. Passive components only consume 1 core, for merging the index.

Determining the need for more mirrors

There are two reasons to add mirrors, redundancy and increased query throughput. Adding the first set of mirrors (passive or active) will allow one or more query servers to be taken off-line with limited impact on end user queries. Taking a look at the SearchBeta Hardware the farm is fully redundant allowing me to patch half of the query servers simultaneously while still servicing end user queries. Building redundancy in the fashion that SearchBeta has with each machine containing 2 active components and 2 passive components (all serving different partitions) and a second machine that is an exact copy with the active/passive components partitions flipped is the implementation that most people will use. This provides the best cost versus benefit since it allows for good query performance during normal operations and still provides service during failure cases. But when a machine is taken out of service the mirrored machine is out of capacity due to memory constraints. Queries don't fail in this case, but latencies do suffer as the passive component starts servicing queries it must first read its portion of the index into memory. Also the common deployment will only have enough memory to cache the active components, so in the failed over state the machine will page the index in and require more memory to be performant.

While adding more partitions to a given index increases the degree of parallelization for a single query; adding more mirrors enables more concurrent queries to be executed. This assumes that the mirrors are marked as active (FailoverOnly == false) and the mirror has 2 dedicated cores. For comparison the query load on SearchBeta is 120 queries per minute. This load causes limited CPU usage and we are not close to needing more throughput capacity. I suspect this will be the common case and the addition of multiple active query components serving the same partition will be rare.

Proof of concept

As you can see sizing a query topology relies heavily on knowing ahead of time how big your index is going to be so you can cache 33% of it in memory. To figure the size of your index you can either estimate the size with the formula in the capacity planning document (Size of Content DB(s) indexed * 0.035) or you can crawl a representative portion of your content and building an estimating formula specific to your data. Generally the 0.035 ratio is assumed to be correct. But your mileage may vary based specifically on the type of content you have. If your content is rife with images and sparse with text then the Content DB will be quite a bit larger than average while the resulting index will be smaller than average. The inverse, content that is verbose with prose, may also cause inaccuracies in the formula. To create your own ratio you will need to calculate the ratio of the index (all files within the cifiles directory on the query server after a Master Merge) to the overall size of your content database (i.e. Index Size/Content Size.)

If you are building out a large scaled environment in the range of 40 million or more documents I would strongly encourage you to look at implementing some type of proof of concept (POC) phase. In this process you should build out a smaller SharePoint 2010 Search farm to index approximately 10 million documents that are representative of your corpora. With the POC you can compare your content to index ratios to confirm how closely your data conforms to average formula we have published.

Thanks for your time, in the next post I plan to delve into scaling of property store and touch on some of the advanced topics for tuning the system for optimal query latencies

Dan Blood

Senior Test Engineer

Microsoft Corp

SharePoint 2010 Search 'Dogfood' Part 2 - Query Health Reports

enterprisesearch — Wed, 09 Jun 2010 16:42:00 GMT

One of my favorite features of SharePoint Server 2010 Search is the health reporting that we added. With the query health reports you can very easily see what is going on with end user query latencies as well as pinpoint were the time is getting spent. The capacity planning document talks about a lot of different guidelines around optimizing the query path, which I plan to detail in future posts, but the main thing to stress is that you should let the reports be your guide as to where to start investigating. If the health reports indicate that query latencies are meeting your goals there is no real reason to proceed further with a performance analysis. Although one benefit of doing an analysis of a healthy system is to provide an understanding of how close you are to capacities so you can plan for future growth.

Out of the box reports

There are two reports titled SharePoint Backend Query Latency & Overall Query Latency. These two reports enable you to diagnose where the costly portion of the query is being executed. The backend report (SharePoint Backend Query Latency) allows you to determine if the time is being spent within the Index (Full-text Query) or the Property store. Moving up the call stack you have the front end (Overall Query Latency) report, with this report you can determine if there is an issue in rendering the UI. For more details about each data series within the reports see the TechNet article Use search administration reports.

SharePoint Backend Query Latency - This report typically has the largest amount of time being spent within the Full-text Index as this is the most time consuming portion of the query.

Overall Query Latency - This report adds Server Rendering and Object Model to the above report (Backend.) All of the time reported in red below is equivalent to what you would see in the backend report.

Query Latency Trend report

We also ship a third query health report. This report is disabled out of the box as it requires additional data to be logged into the Web analytics and health data collection database. This is also disabled under the premise that not all farms, especially content farms, will benefit from having this additional data, but if you are running a Search centric farm I would encourage you to enable this extra data collection. Our SearchBeta has run with this extra data collection enabled all the time and there is no perceived impact to the performance of the system. Just be sure that you segregate the Usage database away from the query system so the additional IO generated does not impact queries.

To enable the extra data collection use the following PowerShell commands:

$app = Get-SPEnterpriseSearchServiceApplication "Your application name"

$app = Set-SPEnterpriseSearchServiceApplication –VerboseQueryMonitoring "True"

Query Latency Trend - The purpose of this report is to provide an all up summary of how your query latencies are doing. The percentiles include time spent in both the front end and backend. The report also includes the system crawl rate to help correlate any trends in query latencies with what is going on in the crawl system.

Utilizing the reports to direct diagnostics

The general process for diagnosing query health is to:

look first at the trending report (Query Latency Trend) to get an idea about how your latencies are trending throughout the day. Do you have a lot of peaks and valleys in the trends or just overall high latencies? If you have peaks and valleys you can look for trends in the peaks over time as well as narrow the time window to focus on a specific peak.

Secondly you would go to either the front end (Overall Query Latency) or backend (SharePoint Backend Query Latency) report to see precisely where all of the time is being spent. Given this you know where to focus further diagnostic efforts: front end, Query Servers or the Property store.

Dan Blood

Senior Test Engineer

Microsoft Corp

SharePoint 2010 Search 'Dogfood' Part 1 - Hardware

enterprisesearch — Thu, 03 Jun 2010 02:12:00 GMT

Hello again, Dan Blood here. To layout the lessons I've learned in hosting Search 2010 I will provide you with a full picture of what the hardware behind SearchBeta looks like. Be aware that the hardware is a little underpowered for the count of items in the index. As such you should not take the hardware I have listed below verbatim and implement your solution on top of it, rather use this hardware and lessons as a starting point. Coupled with the capacity planning document you can then tailor your hardware to your needs and business specific SLA requirements.

The starting point for the hardware behind SearchBeta was to provide a search experience over roughly 60 million items and keep these items freshly crawled within a 24 hour time window. Over time the corpora has grown to include ~72 million items. With 3 Crawl databases the system is almost able to meet a 4 hour freshness target for the majority of the 72 million items.

The ~72 million items are broken out across the following content sources:

Large enterprise collaboration portal (~24 million) -- SharePoint 2010 content

Early adopters (~15 million) -- Portals running the most recent SharePoint codebase

My Sites (~5 million) -- SharePoint 2010 My site content

Europe (~12 million) -- SharePoint 2007 content hosted across a WAN (170ms ping)

Asia (~9 million) -- SharePoint 2007 content hosted across a WAN (200ms ping)

IT Hosted Portals (~3 million) -- Mixed SharePoint 2007/2010 content

Non IT Hosted Portals (~3 million) -- Mixed HTTP and SharePoint 2007 content

BDC (~1.5 million) -- SQL server content crawled via BDC.

The query load of the system is not extreme with peaks of 120 queries per minute. Given the query load and the desire to reduce costs I have implemented query redundancy with an active/passive scheme on each of the 6 query servers. Meaning that each server is hosting 2 active partitions and 2 passive partitions providing full redundancy. This configuration is typical for 60 million items, however, with ~72 million items we are out of capacity. We recommend having enough memory to fit 33% of the index in RAM, with a combined active index size of 106GB we are only able to fit 30% of the index in memory. Because of this 95% of our queries are at 1.1 second during a 24 period, to reach sub-second latencies we would need to meet the 33% guideline.

Hardware & Topology

The hardware for SearchBeta is a 10 box services farm with the following machines:

6 Query Servers

6 Instances of Search Query and Site Settings Service (QP) running on each Query Server

2 Indexers

1 SQL server for the Property & Admin database

1 SQL server for all 3 Crawl databases

There are a few main areas that I would change for SearchBeta hardware if I were to purchase all of the hardware again from scratch:

Add 2 additional Query servers and/or machines with 48GB of RAM to correct our capacity problem. This would allow us to comfortably provide a search experience over 80 million items. The basic guideline is to have approximately 10 million items in active partitions per machine. It is OK to be a little above this, but you must fit 33% or more of the index into memory to have adequate query latency.

Add more memory in the Property SQL machine. However, I would need to look at the cost of purchasing 128GB of RAM versus the cost of purchasing two machines with 64GB. The cost of two machines would only be slightly more expensive (current memory prices) for the initial outlay and it would provide a little more room for growth. The basic guideline is to fit 33% of the Property store into memory.

Add redundancy into the SQL servers. Currently there is no way to patch or cycle the SQL boxes without taking down time for the service.

Query Server machine specs

We initially started with 4 query servers and grew to 6. As a result we have query servers with different clock speeds. This is discouraged as the slower machine will degrade the overall query latency because every query must be executed across each unique index partition.

6 Machines

- 4 - Dell 2950 - 2 quad core 3.00Ghz (8 cores total)
- 2 - Dell 2950 - 2 quad core 2.33Ghz (8 cores total)
- 32GB RAM
- Drives:6-15k SAS 450GB spindles
  - OS - R1
  - Index - R0 4 spindles

Property Database SQL machine specs

1 Machine

- Dell 6850 - 4 dual core 3.2Ghz (8 cores total with hyper-threading disabled)
- 64GB RAM
- Internal drives: 2-15k 148GB; Raid 1 for the OS and SQL.
- 1 - PERC 6/E Raid controller connected to an MD1120

MD1120 24-10k 148GB spindles

Data	Raid Type	Spindles	Space Used Reserved / Used
Property database	Raid 1+0	12	405GB / 232GB
Property database log	Raid 1	2	9GB
Admin database	Raid 1+0	4	12GB / 12GB
Admin log	Raid 1	2	6GB
Temp database & log *8 data files	Raid 1+0	4	114MB / 52MB

Crawler machine specs

2 Machines

- Dell 2950 - 2 quad core 2.33Ghz (8 cores total)
- 8GB RAM
- Drives:2-15k SAs 148GB
  - OS & Product - R1

Crawl Database SQL machine specs

1 Machine

- Dell 6850 - 4 dual core 3.2Ghz (8 cores total with hyper-threading disabled)
- 64GB RAM
- Internal drives: 2-15k 148GB; Raid 1 for the OS and SQL.
- 3 - PERC 6/E Raid controllers each connected to a MD1000

3 - MD1000 45-15k 450GB spindles

Data	Raid Controller	Raid Type	Spindles	Space Used Reserved / Used
Crawl database 1	2	Raid 1+0	6	116GB / 80GB
Crawl database 2	1	Raid 1+0	6	157GB / 56GB
Crawl database 3	1	Raid 1+0	6	70GB / 56GB
Crawl database log 1	3	Raid 1	2	54GB
Crawl database log 2	3	Raid 1	2	95GB
Crawl database log 3	3	Raid 1	2	33GB
Temp database & log *8 data files	3	Raid 1+0	4	6.7GB / 5.7GB

In the coming posts I will dig further into Crawl and Query sides of the system as well as how SQL is utilized. Providing even further details about how to monitor the running system and what areas to look at to see if the system is reaching capacity.

Dan Blood

Senior Test Engineer

Microsoft Corp

Server Name Mapping and Alternate Access Mapping (AAM)

enterprisesearch — Sat, 29 May 2010 01:09:00 GMT

Alternate Access Mapping (AAM) and Server Name Mapping are two features SharePoint provides to alter the URLs returned in search results - a common practice used by many companies for their internal and external users. Both features are available in both the out of the box search in SharePoint 2010 and in the new FAST Search Server for SharePoint 2010.

This post is intended for people with working knowledge about the two features who want to know when to use which. If you need to brush up on your basic understanding about the two features, here are a couple good sources for your convenience:

· Alternate access mapping: http://technet.microsoft.com/en-us/library/cc261814.aspx

· Server name mapping: http://technet.microsoft.com/en-us/library/cc164184.aspx

Although Server Name Mapping and Alternate Access Mapping achieve seemingly similar results, they work independently, addressing different problems, and should not be used together. I’ve listed the similarity and differences below:

Server Name Mapping

· Is designed for file share and http content.

· Allows you to map any server name to anything you like as long as the name you mapped to actually points to the same set of pages. For example, you might have a Web site with a real access URL http://foo that you want show as http://microsoft.com ; or an internal file share server \\foo and you want to use \\microsoft instead. By setting up Server Name Mapping, your crawler will use http://foo or \\foo for indexing, but your users will only see http://microsoft.com, or \\Microsoft .

· Requires a full crawl for the mapping, once set up, to be applied.

· Search results will always use the new name for all users.

· Settings overwrite the AAM setting for the same results, if you use them together - which you shouldn’t.

· Is defined by the search admin for each SSA (search service application)

Alternate Access Mapping

· Is designed for SharePoint content.

· Allows you to modify results URLs based on the access URLs for a site. So, for the same result page, a user accessing from an internal URL will see results with URLs matching the internal site, and a user accessing from an external URL may get the same result set but with URLs matching the external site. For example, for the same set of SharePoint content, internal users use http://server to access the site, all the URLs they get in search results are prefixed with http://server/... ; external users use http://www.microsoft.com, all the search results they see are prefixed with http://www.microsoft.com.

· Does not require a crawl for the settings to take effect.

· Generates results URLs based on how the site is accessed.

· Is set per farm, so if you have more than 1 search service application in your farm, all of them will use the AAM setting.

The combination of Search Server Mapping and Alternate Access Mapping offers a lot of flexibility in managing URL mappings in search results. I hope this post helps to clarify how these features work and compare. If you have any questions or observations, please don’t hesitate to post your comments here.

Ying Tao, Test Engineer, SharePoint Productivity Search

SharePoint 2010 Search - Experiences and Best Practices from the Microsoft ‘Dogfood’ Implementation

enterprisesearch — Mon, 10 May 2010 21:54:00 GMT

Hello, Dan Blood here with another series of blog postings about running SharePoint Server 2010 Search. Some of you may be familiar with a similar set of postings that I wrote for MSS or SharePoint Search 2007. The following set of postings will be targeted explicitly at SharePoint 2010 Search and are not applicable to the previous release. Similarly these posts do not contain any data specific to FAST Search Server for SharePoint 2010.

I am very excited about the changes and new features in SharePoint 2010 Search. There are a plethora of things that make building and running the system at scale much easier.

The scale limits have dramatically increased from 2007.
The system has been componentized to allow scaling crawls and queries separately.
The index now has the ability to be partitioned into smaller more manageable pieces and spread out across multiple machines.
Crawlers have become stateless and no longer house the entire index. Resulting in reduced hardware requirements for the crawlers.
You can allocate more than one machine to crawl the content for improved crawl freshness.
The system is much more transparent, providing admin's with information about what is going on under the hood.

Over the last 3 years I have been managing our internal "Dogfood" deployment (SearchBeta) that mirrors the enterprise search deployment within Microsoft and is used for product testing and validation. This deployment provides search over ~72 million items and services a peak of 120 queries per minute. I've also had help from Hernando Silva in maintaining a separate DogFood system for the Office organization. This is a divisional portal providing a search experience over ~7 million items. Hernando and I will detail these two systems in following posts to provide you with insight into the hardware choices. These efforts have helped us validate the product utilizing real world data for both large & small scale deployments. I believe that our experiences will provide you, our customer, with a valuable set of best practices and lessons about how to run SharePoint 2010 Search in production.

While I am working on the next post I would encourage you to read the following documents as starter for what the new system now looks like:

The TechNet Enterprise Search site provide some very useful getting started & what's new content
Design Search Architectures for Microsoft SharePoint Server 2010
SharePoint 2010 Search Capacity planning document

Dan Blood

Senior Test Engineer

Microsoft Corp

Business Data Search in SharePoint 2010: Crawling associated external content types

enterprisesearch — Tue, 09 Mar 2010 18:48:00 GMT

Business data associations are widely used to fetch related external content (ex. data from two sql tables connected by a foreign key or an email that contains attachments) and display it in SharePoint webparts. SharePoint Search 2010 ships with new features that enable crawling of Business Data Connectivity associations. This post describes how to use the search specific metadata properties for crawling associations and the different user experiences that can be enabled. For a primer on how to define associations, see http://msdn.microsoft.com/en-us/library/ms561493.aspx.

From the search perspective, the source external content type of an association is referred to as the Parent external content type. Search can crawl external content types associated with the parent in two ways, as attachments or as child external content types. Let's look at what this means for the user experience, incremental crawl and deletes.

User experience

A child external content type has its own result URL and profile page (if one is created). This is the URL that will be displayed if a user searches for a keyword in the child external content type data. An attached external content type on the other hand, does not have its own URL. If the user searches for a keyword in the content of the attached external content type instance, the URL for the parent external content type will be displayed. This URL can be set to the profile page URL of the parent. A profile page for the parent external content type will display all the fields of the attached external content type exposed by the association navigator.

Incremental crawl

For a timestamp based incremental crawl if the timestamp of the child external content type instance changes, it will be re-crawled and updated.

The timestamp of the parent external content type instance is interpreted as the timestamp of attached external content type instance. So any change in the attached external content type instance will be picked up by the incremental crawl only if the timestamp of the parent external content type instance changed.

Processing deletes

When processing deletes, if the parent external content type is deleted, search will delete attached and child external content types from the index.

Crawling an associated external content type as attachment

To mark an association to be crawled as attachment, add the AttachmentAccessor property to the Association as shown in the sample below. Search does not care for the value of the property.

<Properties>

<Property Name="ForeignFieldMappings" Type="System.String">....... </Property>

<Property Name="AttachmentAccessor" Type="System.String">x</Property>

</Properties>

<SourceEntity Namespace="Foo " Name="Parent" />

<DestinationEntity Namespace="Foo" Name="Attached External content type" />

</Association>

Crawling an associated external content type as child external content type

To mark an association to be crawled as child external content type, add the DirectoryLink property to the Association as shown in the sample below. Search does not care for the value of the property.

<Properties>

<Property Name="DirectoryLink" Type="System.String">x</Property>

</Properties>

<SourceEntity Namespace="Foo" Name="Parent" />

<DestinationEntity Namespace="Foo" Name="Child External Content Type" />

</Association>

Nitin Ved, SDET | Microsoft Enterprise Search Group

Innovation on Linux and UNIX

enterprisesearch — Thu, 04 Feb 2010 22:19:00 GMT

Innovation is at the heart of our enterprise search strategy, and a commitment to innovation is what brought FAST and Microsoft together. When we announced the acquisition two years ago, we said that we were committed to cross-platform innovation—that we’d “continue to offer stand-alone versions of ESP that run on Linux and UNIX,” and that we would provide updates to these versions to address customer concerns and add new features. Over the last two years, we’ve done just that. We’ve continued to sell, support, and update the Linux and UNIX versions of FAST ESP, and we’ve designed the next wave of FAST products (scheduled for release in the first half of calendar year 2010) to include a cross-platform search core that has been extended to take advantage of web services and support mixed-platform deployment models.

With our 2010 products scheduled for release in a few months, we’ve just started to plan for our next wave of products. As a part of that planning process, we have decided that in order to deliver more innovation per release in the future, the 2010 products will be the last to include a search core that runs on Linux and UNIX.

Many of our customers run FAST ESP on Linux and UNIX today, and we recognize that our future focus on Windows means change. To ease the transition, we’re investing in interoperability between Windows and other operating systems, reaffirming our commitment to 10 years of support for our non-Windows products, and taking concrete steps to help customers plan for the future.

§ We will always interoperate with non-Windows systems on both the front- and back-end. Our search solutions will crawl and index content stored on Windows, Linux, and UNIX systems, and our UI controls will work with UI frameworks running on any operating system.

§ We will support ESP 5.3—the cross-platform search core in the 2010 products—for 10 years as per our standard support policy (5 years mainstream support and 5 years extended support). Non-Windows customers on ESP 5.3 can continue running their core on Linux and UNIX and add Windows-only innovations or cloud-based services by using a mixed-platform architecture.

§ We’re introducing a Customer Upgrade Program that will help customers evaluate our hosted solutions and/or a Windows-based deployment. The program will help customers assess level of effort, plan for the project, and implement the upgrade. (Customers should contact their account representative for more details.)

When FAST was founded back in 1997, we were told that it was too late to start a search company. The prevailing wisdom back then was that search was already a commodity: Verity had won the enterprise and AltaVista had won the web. More than ten years later, it’s clear that we’re just getting started. Today I actually see more opportunities than ever for search to change the world in profound ways—and I believe that innovation is the magic that will make it all happen. Although I understand that focusing on Windows will be a hard change for some of our customers, I’m convinced that it’s the right thing to do because it will accelerate our rate of innovation. By focusing on Windows we’ll be able to tap into a valuable set of competencies and assets across Microsoft and deliver better technologies more efficiently to our customers. I recognize that if you run FAST ESP on Linux and UNIX today, you’ll likely have a lot of questions over the next few days and weeks. There’s no immediate action required as a result of this announcement—and I expect that most of you will stay with your current deployments for some time. But I encourage you to contact your account representative to talk about what this means for you and to learn more about what we can do to help. We’re committed to working with you through the transition and look forward to partnering with you for success now and in the future.

Bjørn Olstad

CTO, FAST

Distinguished Engineer, Microsoft

Regular Expressions Support in SharePoint 2010 Crawling

enterprisesearch — Thu, 21 Jan 2010 20:52:00 GMT

Search admins often need to omit from a crawl files that match a certain pattern. E.g.:

· In a bank, file names starting with SSN

· In a business site, files names with credit card number

· URLs having specific value of a certain parameter of an aspx file

· etc..

The usual solution is to allow admins to create “crawl rules” that restrict crawlers from following specific links. The most basic crawl rule specifies a complete URL for the file to be crawled, which requires the admin to create as many rules as there are files in their repository. A more practical solution often implemented involves the use of the wildcard character: “*”. This character matches everything, so admins can create a rule using the wildcard to include (or omit) all files under a particular folder or path:

\\myfileserver\myclientsfolder\*

This works if all the files are located neatly in one folder, but what if they are spread across the repository (or Web site)? This is the problem that is solved by using regular expression (RegEx) syntax.

The SharePoint Solution

In SharePoint 2007, the wild card operator “*” is the only operator supported in crawl rules for matching characters. As mentioned, it is a brute force operator that matches everything. Wildcard-only rules do not provide the admin the flexibility to, for example, recognize and omit URLs that contain Social Security Numbers, or that have an aspx parameter with a specific value.

SharePoint 2010 includes some new capabilities in this area. The default behavior of crawl rules in SharePoint 2010 is the same as it was in SharePoint 2007, but with SharePoint Search 2010, administrators can create crawl rules to include or exclude URLs that match regular expressions. To enable regular expressions, the admin need only select the check box on the Crawl Rules creation UI as shown in the image below.

Regular Expression Operators

The table below lists and describes the regular expression operators that are supported for crawl rules in SharePoint 2010:

Operator	Symbol	Description	example Rule	Will match	Won’t match
Group	()	Characters can be grouped using round brackets. Any operator applied on it will be applied on the group.
Match any character	.	This operator matches any character. It does not match with NULL.	http://mysite/page.ht..	http://mysite/page.html	http://mysite/page.htm
Match zero or one	?	It allows the expression to not exist in the target address or can have only one repetition.	http://mysite/page(1)?.html	http://mysite/page.html AND http://mysite/page1.html	http://mysite/page11.html
Match zero or more	*	It allows the expression to not exist in the target address or can have any number of repititions.	“http://mysite/page(1)*.html	http://mysite/page.html AND http://mysite/page111.html	http://mysite/pag.html
Match at least one	+	It requires the expression on which it is applied to exist in the target address at least once.	http://mysite/page(1)+.html	http://mysite/page1.html AND http://mysite/page111.html	http://mysite/page.html
Exact count	{num}	This operator is denoted by a number inside “{}”, e.g. {5}. It restricts the expression on which it is applied to have exactly the specified number of repetitions in the target address.	http://myfiles/(9){4}-(0){2}.html	http://myfiles/9999-00.html	http://myfiles/999-00.html
Minimum count	{num,}	This operator is denoted by a number inside “{}” followed by a "," e.g. {5,}. It restricts the expression on which it is applied to have at least the specified number of repetitions in the target address.	http://myfiles/(9){4,}-(0){2}.html	http://myfiles/9999-00.html AND http://myfiles/99999-00.html	http://myfiles/999-00.html
Range count	{num1,num2}	This operator is denoted by 2 numbers inside “{}” separated by a "," e.g. {5,8}. First number defines lower limit and second number defines the upper limit. It restricts the expression on which it is applied to have any repititions in the URL between num1 and num2. A valid rule will always have num1 < num2.	http://myfiles/(9){4}-(0){2,3}.html	http://myfiles/9999-00.html AND http://myfiles/9999-000.html	http://myfiles/9999-0000.html
Alternation	\|	This operator is applied on two expressions and it matches ONLY one of the two expressions.	file://myshare/((folder1)\|(folder2))/.*	\\myshare\folder1\<any files> OR \\myshare\folder2\<any files>	\\myshare\folder1folder2\<any files>
List	[<list of chars>]	This operator is denoted by a list of characters inside “[]”. It matches with any of the characters which are specified in the list. Admin can specify a range of characters by using "-" operator in it.	http://testhost/test[1-3].htm	http://testhost/test1.htm OR http://testhost/test2.htm OR http://testhost/test3.htm	http://testhost/test.htm

Using RegEx Operators in Crawl Rules

Once you understand the RegEx operators above and how to enable them in the crawler, there are only a couple other things you need to keep in mind:

Protocol part

Regular expression operators cannot be used in the protocol part of the URL. This means, for example, the following RegEx rule cannot be created:

.*//www.microsoft.com/.*

If you try to create a rule like this, the system will add http:// in the beginning and thus make “.*” as the second part of the URL. The resulting rule in this case will be:

http:// .*//www.microsoft.com/.*

which may not be what you intended.

Case sensitive comparison

RegEx rules are case insensitive by default. In order to allow a rule to do case sensitive matching of a URL, the administrator should select the “Match case” check box in the rule creation UI as shown below:

If the “Match case” checkbox is selected, the crawler will preserve the case of matching URLs during the crawl. In the example above, the rule will match: http://test/AbC123.html and WILL NOT match to http://test/Abc123.html.

This feature comes in handy when SharePoint is used to crawl web sites hosted on Unix based web servers, which are case sensitive.

Examples

Here are some interesting examples demonstrating the usefulness of Regular Expression in crawl rules:

Rule	Description
\\myshare\.*	Match everything under the share “myshare”
file://.*/[0-9]{4}-[1,2]-[a-z]{4}.docx	Match all the links with file names having the following pattern: <4 digints>-<1 or 2>-<4 characters>.docx
\\myshare\((folder1)\|(myfolder))\.*	Match all files in folder1 or myfolder in \\myshare
http://mysite/myasp.aspx[?]param1=value	Specify a regex operator: "?" in this case, in regex rule.
http://mysite/myasp.aspx[?]param[12]=.*	Match all links pointing to myasp.aspx with either param1 or param2 specified.
http://site/..aspx[?]category=1&subcategory=.	Match all aspx links that have a specific parameter value and ignore the value of second parameter
file://clientsdata/[0-9]{3}-[0-9]{2}-[0-9]{4}.*	Match all files that start with Social Security Number

Syed Anas Hashmi | SDET | Microsoft Enterprise Search Group