Microsoft Enterprise Search Blog : Search Server

Microsoft positioned in the Leaders Quadrant of the 2008 Information Access Magic Quadrant

enterprisesearch — Fri, 31 Oct 2008 00:33:00 GMT

We’ve got great news to share! Last month, Gartner published the 2008 Magic Quadrant for Information Access Technology, and Microsoft was positioned in the Leaders Quadrant. Since the completion of the acquisition, we’ve worked incredibly hard to communicate and demonstrate a combined vision and strategy to our customers and partners. It’s good to know we’re heading in the right direction!

When I talk with customers about search, it’s clear that organizations have very different needs. In fact, many people tell me that even within an organization the one-size-fits-all approach just doesn’t work. So over the last year, we’ve announced some bold moves designed to create a compelling portfolio of search applications. With the addition of Search Server Express and the acquisition of FAST, we now have a product line-up designed to meet a broad range of business needs:

Some departments or small organizations need search that is quick and easy to set up; we offer Microsoft Search Server Express as a free download so that you can get it up and running in about 30 minutes. We’re excited to see customers like St. Jude Medical and Urbis having quick successes with Express. We’re also seeing partners, such as StartReady, build solutions around Search Server Express to create a search appliance.
Many organizations need search as an integral part of a business productivity infrastructure; Search in Microsoft Office SharePoint Server is integrated with other key SharePoint productivity workloads such as portals, collaboration, ECM, business processes and BI. Customers like McCann Worldgroup and Jones Lang LaSalle are all deriving productivity increases with better search in SharePoint. In particular, both companies are promoting collaboration and leveraging in-house experts with people search enhanced by user profiles in MySites.
Some organizations face business problems that demand high-end search; FAST ESP offers best-in-class search with extreme scalability, query performance, and other advanced capabilities for sophisticated customer-facing or inside-the-firewall applications. For example, Aerotek and TEKsystems, two of the world’s largest staffing companies, deliver job searching to more than 1.3 million users. In more than 164 million queries, greater than 99.5% of query results came back in less than 2 seconds. For inside-the-firewall productivity, they index more than 10 million complex candidate records with low latency during high volume index updates. We’re also excited to see Pfizer pushing the envelope with an Enterprise Collaboration Framework driven by FAST ESP on top of SharePoint

While our “Leaders Quadrant” position in the Magic Quadrant is an important milestone, we still think of this as the very beginning of our journey. We’re continuing to combine our deep technical expertise with our broad reach to deliver exciting innovations to the market – so you can and should expect great things to come. Stay tuned!

Kirk Koenigsbauer
General Manager,
SharePoint Business Group

Magic Quadrant for Information Access Technology (Gartner Research, Sept. 30, 2008) Microsoft is positioned in the Leaders Quadrant of Gartner, Inc.'s 2008 Magic Quadrant for Information Access Technology. This report assesses vendors with capabilities that go beyond enterprise search to encompass a range of technologies. Their capabilities include search; federated search, content classification, categorization and clustering; fact and entity extraction; taxonomy creation and management; information presentation (for example, visualization) to support analysis and understanding; and desktop search to address user-controlled repositories in order to locate and "invoke" documents, data, e-mail and intelligence.

The Magic Quadrant is copyrighted 2008 by Gartner, Inc. and is reused with permission. The Magic Quadrant is a graphical representation of a marketplace at and for a specific time period. It depicts Gartner's analysis of how certain vendors measure against criteria for that marketplace, as defined by Gartner. Gartner does not endorse any vendor, product or service depicted in the Magic Quadrant, and does not advise technology users to select only those vendors placed in the "Leaders" quadrant. The Magic Quadrant is intended solely as a research tool, and is not meant to be a specific guide to action. Gartner disclaims all warranties, express or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

Taking People Search on the Road….

enterprisesearch — Thu, 16 Oct 2008 20:33:53 GMT

In another great blog post Matt McDermott walks you through the steps of enabling SharePoint’s people search capability on a mobile device with the end results looking something like this;

The post is here;

http://blogs.catapultsystems.com/matthew/archive/2008/09/27/mobile-people-search.aspx

Richard Riley
Senior Technical Product Manager
Microsoft Corp.

SharePoint Image Search

enterprisesearch — Sat, 20 Sep 2008 01:03:43 GMT

Matthew McDermott, a SharePoint MVP, has written a great 4 part blog post on how to make SharePoint 2007 search (and Search Server) render image results in a way that looks very similar to http://images.live.com.

Not only does this make searching images much easier, it’s also a very thorough step-by-step tutorial on how to customize results using the built in Web Parts and XSL – it’s well worth a read.

SharePoint Image Search (Part 1)

SharePoint Image Search (Part 2)

SharePoint Image Search (Part 3)

SharePoint Image Search (Part 4)

The end result makes SharePoint Image results look like the screencap below.

Richard Riley
Senior Technical Product Manager
Microsoft Corp.

Announcing Faceted Search v2.5

enterprisesearch — Wed, 13 Aug 2008 00:50:22 GMT

Starting Faceted Search 2.5, the solution relies on Microsoft Enterprise Library to address common software requirements in caching, logging, exception handling, policy injection etc., etc. More importantly, the 2.5 is a ground breaking release that is setting new targets for the Faceted Search. So, what’s new?

New Features

1. Caching – dramatically improves performance and decreases the load on the search engine

The solution uses 2 mechanisms for manageable cache: quick and long. I built the caching logic on assumption that user knows what he/she is looking for. The Search Facets web part will cache original result set and use it for the search refinement, paging and other postbacks. If the initial result set doesn’t provide full coverage of the search, the smart 2^nd thread will run against real-time data providing adjustment to the cached match.

2. Synchronization with Core Search Results web part

The MOSS search is adjusted by several parameters that designer can set for the Core Search Results web part itself. These include remove duplicates, enable trimming, permit noise words. When you drop the Search Facets web part to the search results page, it will find the Core Search Results, read its parameters and sync the search query parameters to exactly match ones used by the Core.

3. Support for advanced search

It was the most wanted feature since Faceted Search 1.0. With 2.5, the Facets are rendered for advanced search although do not extend yet to ranges. The functionality is accomplished by extending SearchQuery structure to accommodate POST requests and sync back to GET query.

4. Match of search counters

This release introduced an updated search syntax that is design to provide matching counters to the core search. In fact, the new search query is using both KeywordQuery and FullTextQuery through the use of generics.

public class GenericQuery<T> : IDisposable where T : Query
{
    private EventHandler _customLogic;

    public ResultTableCollection Execute(EventArgs args)
    {
        _customLogic(_query, args);
        return _query.Execute();
    }

    ...
}

Additionally, the WHERE clause of the search query was modified to provide closer match to the Core counter.

5. Introducing Parent-Child relationships

By design, the facets can support only 2 levels. This release extended the Facets schema to allow management of the nested layers. That eases the pain of displaying complex hierarchies such as geography, or org chart etc. Parent-Child relationship can be set by facet name and facet value, or just by facet name.

<Column Name="BDCCity" DisplayName="City" ParentName="BDCState" />
<Column Name="BDCState" DisplayName="State" >
  <Mappings>
    <Mapping Match="Alberta"  ParentName="BDCCountry" ParentValue="Canada"/>    
    <Mapping Match="Manitoba" ParentName="BDCCountry" ParentValue="Canada" />
    <Mapping Match="Ontario"  ParentName="BDCCountry" ParentValue="Canada"/>
    <Mapping Match="Quebec"   ParentName="BDCCountry" ParentValue="Canada"/>
  </Mappings>
</Column>

In the configuration above, the City facets will display only after the user chose the State. The State itself will match the country of origin.

6. Extending search to logical “OR” queries

Original facets always represent “AND” queries. That implies ability to narrow the search results by adding extra criteria. In this release I prototyped the way to expand the search by adding additional matches to the criteris. This in fact resulted in rewamped the Bread Crumbs UI. Proviuded now out-of-the-box support for languages is a good example of how “OR” queries empower the search.

7. Simplified web part properties

The 2.5 release is friendly to modifications of the web part properties. I have all properties classified and broken down to groups for each of the web parts.

8. Other

There are lots and lots of numerous fixes and enhancements, including improved security validation, code refactoring, extending facet sorting, support of quoted search and duplicates etc., etc.

What’s next

It’s my privilege to say that we have a team now that helps to shape new releases and brainstorm the furutre of the Faceted Search. In present we are looking at AJAX and SilverLight and hopefully you’ll start seeing more and more power of Facets in the near future.

Leonid Lyublinski
Senior Consultant
Microsoft Consultancy Services

Announcing: Availability of Infrastructure Updates

enterprisesearch — Tue, 15 Jul 2008 10:08:13 GMT

As announced on the SharePoint Team blog this morning we released to web three new important updates that affect SharePoint Server 2007, Windows SharePoint Services 3.0, Project Server 2007, Search Server 2008, Search Server 2008 Express and Project Professional 2007.

The Infrastructure Update for Microsoft Office Servers (KB951297) (Download X86, Download X64) is particularly important from a Search and SharePoint Server 2007 perspective as it contains the new Enterprise Search features that were shipped in Search Server 2008 and Search Server 2008 Express that were are not already in SharePoint Server 2007; this includes Federated Search capability, a unified administration dashboard and several Search core platform performance updates.

For an overview of the new federation features please check out this short video which covers how to configure a federated location and configure one of the new federated search Web Parts to show results from that location.

There’s also a growing number of articles on TechNet and MSDN that cover configuring and troubleshooting federation and extending federation with Federated Search Connectors.

The screen capture below shows how federated search results show up on a results page – the results on the right hand side and top left are federated results, the ones at the bottom left are from the local index.

The new Search Administration Dashboard consolidates all of the Search related admin activities into a single place – there’s also some new functionality in the dashboard (There’s greater granularity for content source crawl history and a convenient list that shows currently running crawls and durations for example) and it makes the Search Administrators job much easier by keeping everything close at hand.

The UI looks like the screen capture below:

The update leaves the old Search admin pages intact, the links to them stay in Central Admin (Along with a new link to the new dashboard) so if you’ve made any changes to them or just prefer to use the existing admin pages you’re free to do so.

The other changes are all under the hood and improve Index and Query performance as well as fixing a few bugs. Check out the KB articles below for more details.

Description of the Infrastructure Update for Microsoft Office Servers (KB951297)

Fixes Included in the Infrastructure Update for Microsoft Office Servers (KB953750)

Please read this post on the SharePoint Team blog and the installation instructions thoroughly before you install the Infrastructure Update for Microsoft Office Servers (KB951297) and the Infrastructure Update for Windows SharePoint Services 3.0 (KB951695) on SharePoint Server 2007 or Search Server 2008.

Install the Infrastructure Update for Microsoft Office Servers (Office SharePoint Server 2007)

Install the Infrastructure Update for Microsoft Office Servers (Search Server 2008)

Finally, if you’re wondering why the Infrastructure Update for Microsoft Office Servers (KB951297) and the Infrastructure Update for Windows SharePoint Services 3.0 (KB951695) applies to Search Server 2008 then you’re probably not alone!

There’s a very good reason for both – Search Server 2008 and Search Server 2008 Express are built on the Windows SharePoint Services 3.0 platform (hence the need for the Infrastructure Update for Windows SharePoint Services 3.0 (KB951695)) and secondly the Search features are from SharePoint Server 2007 (hence the need for the Infrastructure Update for Microsoft Office Servers (KB951297)). The latter update includes a few bug fixes since Search Server 2008 and Search Server 2008 Express launched.

We strongly recommend that you install the updates that apply to you as soon as your patching and maintenance schedules permit.

Richard Riley
Senior Technical Product Manager
Microsoft Corp.

Announcing: conceptClassifier for SharePoint

enterprisesearch — Tue, 08 Jul 2008 00:41:07 GMT

conceptClassifier for SharePoint adds automatic document classification and taxonomy management to Microsoft SharePoint and works without the need to build another search index. It is installed as a set of Features that, when activated, cause new columns to be displayed in the document library listings and new menu options appear that allow authorised users to edit the automatically generated metadata, if required.

Adding Taxonomy navigation to SharePoint

Classification results are saved directly into SharePoint Properties where Microsoft Enterprise Search can utilise the metadata for enhanced searching, such as faceted search or results filtering.

The accuracy of the automatic classification is driven by the underlying technology which is based on compound term processing. This means that the classification engine performs its matching using multi-word concepts rather than simply looking for selected keywords or phrases. Taxonomy creation and maintenance is a simple process and is conducted using natural language rules, making it much simpler and quicker than alternative approaches.

More information about conceptClassifier for SharePoint can be found here:

http://www.conceptSearching.com

and a SharePoint demonstration can be seen here:

http://moss.conceptSearching.com

John Challis
CTO
Concept Searching Limited

Announcing: SharePoint Web Parts for FAST ESP

enterprisesearch — Fri, 20 Jun 2008 20:06:00 GMT

It’s been around 45 days since the acquisition of FAST Search and Transfer closed and we’re moving quickly to provide interoperability for Microsoft customers between FAST ESP and Microsoft SharePoint Server.

The first deliverables from this work are a set of FAST ESP Search Web Parts for quickly integrating results from FAST ESP into SharePoint Server 2007 and a FAST ESP Search site template.

Using these Web Parts and Site Template SharePoint administrators will be able to quickly and easily build FAST ESP-based search sites inside SharePoint 2007 by simply dropping in and configuring the appropriate components.

The Web Parts and Site Template are available as a free download (both compiled code and source code) from CodePlex at www.codeplex.com/espwebparts and are part of the Search Community Toolkit.

Some of the FAST ESP search capabilities that can be exposed within SharePoint Server 2007 using these Web Parts include:

• Search Box Web Part -- Search box for query term submission and includes “did you mean” functionality for query correction

• Result List Web Part -- Displays search results and supports sorting, pagination, and navigator-based filtering

• Navigator Web Part -- Displays dynamic navigators that profile search results across a set of pre-defined dimensions and allow users to refine the search through navigation clicks

• Breadcrumb Web Part -- Displays the search term(s) and list of navigators used to obtain the current result set

The FAST ESP Web parts are designed to be open and extensible, and we’re actively encouraging customers and partners to download them, customize them to align with their branding and extend them to fit their search and user experience requirements.

Expect the features, functionality and range of ESP Web Parts to grow through contributions from the search developer community as well as further contributions from the FAST & Microsoft Search Team!

FAST & Microsoft Search Teams.

Indexing Exchange Server 2007 Public Folders

enterprisesearch — Sat, 07 Jun 2008 03:56:42 GMT

I've had several questions recently about how to index Exchange Server 2007 Public Folders with SharePoint Server 2007.

Unfortunately with the RTM versions of both products it's not actually possible due a couple of issues with both Exchange Server 2007 and SharePoint Server 2007.

The good news however, is that everything is back in working order if you install Exchange Server 2007 Service Pack 1 *and* your SharePoint Server also has Service Pack 1 installed.

The RTM versions of Search Server 2008 and Search Server 2008 Express are unaffected (As they already include the Service Pack 1 changes), so providing Exchange Server 2007 has Service Pack 1 installed they will both work.

None of this affects Exchange Server 2003 which works with the RTM versions SharePoint Server 2007, Search Server 2008 and Search Server 2008 Express and SharePoint Server 2007 with SP1.

Hopefully this stops people scratching their heads...

Richard Riley
Senior Technical Product Manager
Microsoft Corp.

Introducing Protocol Handler.NET

enterprisesearch — Thu, 05 Jun 2008 09:13:27 GMT

"Protocol Handler.Net is a set of .Net wrappers for the protocol handler interfaces that enable developers to create and deploy protocol handlers for SharePoint search and Search Server.

Developers can index data and documents from any system they can connect to.

Much of the complexity and time around the development of protocol handlers, such as COM interoperability, are reduced and hidden in the wrappers themselves letting developers just concentrate on code to connect to a content source and pull data.

Protocol Handler.Net makes it possible to develop protocol handlers in C# or VB.Net and simplifies the handling of security, metadata, streaming content, deployment and management just to name a few things. It also comes with a help system and sample project to further help developers."

Big thank you to to Chris Gomez from http://www.FastSharePoint.com for creating and sharing these tools and samples!

They are available now on CodePlex at http://www.codeplex.com/phdotnet and are part of the Search Community Toolkit

Announcing: Release to Web of Documentum and FileNet Indexing Connectors

enterprisesearch — Wed, 28 May 2008 03:45:46 GMT

Today marks the release of 2 new Microsoft Enterprise Search Indexing Connectors (formally known as Protocol Handlers) for EMC Documentum 5.3 (Service Pack 4) and IBM FileNet P8 3.5.1 or 3.5.2.

The connectors are compatible with the 32bit English Language versions of SharePoint Server 2007 (Service Pack 1), Search Server 2008 and Search Server 2008 Express, and are available as FREE downloads from;

Enterprise Search Indexing Connector 2008 for EMC Documentum

Enterprise Search Indexing Connector 2008 for IBM FileNet

Installation and configuration documentation is included in the download and the release notes are available here;

Release Notes for Indexing Connector 2008 for EMC Documentum

Release Notes for Indexing Connector 2008 for IBM FileNet

A couple of overview videos to get you up and running with the connectors quickly are available through the links below;

Overview Video - Installing and configuring the EMC Documentum Connector

Overview Video - Installing and configuring the IBM FileNet Connector

Both of these connectors are fully supported through your existing service contract with Product Support Services or through the regular pay per incident channel.

Introducing the ExportCrawlLog STSADM Command Extension

enterprisesearch — Tue, 27 May 2008 06:46:21 GMT

In versions of SharePoint prior to MOSS 2007 each time a crawl was executed, a new group of log messages were stored to the database. Also, the name of the log itself was changed in the documentation and the user interface. Formerly this log was known as the Gatherer Log, but it is now called the Crawl Log.

When troubleshooting problems with the crawl of a particular content source it was (and still is) sometimes useful to compare and contrast the messages logged between one crawl and the next. In MOSS 2007, the storage of the crawl log messages has been optimized/minimized such that only the most recent message for a given URL is stored in the database. As a consequence, the results from a prior crawl are overwritten by results from subsequent crawls. In other words, you can only ever see the most recent log message for a given URL.

This is where the STSADM command extension “ExportCrawlLog” comes in. The motivation for preparing this tool is to provide a way to make a “snapshot” of the Crawl Log information at a point in time to facilitate post-mortem analysis of crawl problems. As a bonus, in addition to extracting crawl log detail, it also provides some summary reporting features. The goal of the tool is to provide a means of gathering data by which you can track and manage the health of your index over time. For instance you could setup a scheduled task to run this command once a day and generate summary reports that can provide data for trend monitoring.

ExportCrawlLog uses only the published APIs of the SharePoint Object Model and must be run on the index server of your SharePoint Farm. ExportCrawlLog is available as source code on Codeplex at http://www.codeplex.com/ExportCrawlLog and is part of the Search Community Toolkit.

Please use the Discussion tracking and Issue tracking features of Codeplex to offer your feedback.

Larry Kuhn
Architect
Microsoft Consulting Services.

Understanding Total Hits & Paging in the MOSS 2007 Search API

enterprisesearch — Thu, 22 May 2008 21:45:06 GMT

One of the more discussed topics I’ve seen (and struggled with myself) is around the concept of obtaining the total number of hits in a search results when working with the MOSS API. For instance, when I search for “sales forecast” in my SharePoint site, I want to not only see a set of paged results, 10 hits per page, but also see that my search found 127 matches. Those of you who’ve worked with the SharePoint Search Web Parts know this is a piece of cake using the Search Core Results, Search Paging and Search Statistics Web Parts.

But what if you need to roll your own solution? How can you get the same data out of your search query using the MOSS 2007 Search API? At first this can be a bit tricky but this post will hopefully show you how to knock it down to being a trivial task.

Executing a search query against the SharePoint API has you working with two objects that implement the abstract class Microsoft.Office.Server.Search.Query.Query: Microsoft.Office.Server.Search.Query.KeywordQuery and Microsoft.Office.Server.Search.Query.FullTextSqlQuery. The former KeywordQuery is useful for simple queries whereas the latter FullTextSqlQuery is much more powerful. Both implement the Execute() method which executes the defined query and returns back a collection of results as type Microsoft.Office.Server.Search.Query.ResultTableCollection. Using this object, you can get the specific results you are interested in. For instance to get the relevant results use the following to get an instance of a specific Microsoft.Office.Server.Search.Query.ResultTable:

using (FullTextSqlQuery query = new FullTextSqlQuery(SPContext.Current.Site))

{

query.QueryText = "SELECT Rank, Title Url FROM Scope() WHERE FREETEXT(defaultproperties,'sales proposal') ORDER BY Rank Desc",

ResultTableCollection results = query.Execute();

ResultTable relevantResults = results[ResultType.RelevantResults];

// do work with the results

}

Simple enough, but the project requires much more than that as usual. What we need to do is page the results to show only 15 items per page. No problem… let’s just modify that query a bit to set the Query.StartRow & Query.RowLimit properties of the query to say what page we’re on and tell SharePoint how many results we want to get back. Take for instance if we’re on page 2 of the results… we want to start with the 16^th hit as 1-15 were on page 1:

using (FullTextSqlQuery query = new FullTextSqlQuery(SPContext.Current.Site))

{

query.StartRow = 16;

query.RowLimit = 15;

query.QueryText = "SELECT Rank, Title Url FROM Scope() WHERE FREETEXT(defaultproperties,'sales proposal') ORDER BY Rank Desc",

ResultTableCollection results = query.Execute();

ResultTable relevantResults = results[ResultType.RelevantResults];

// do work with the results

}

Again… pretty straight forward. Now is where it gets a bit tricky. You need to show links to provide paging… but in order to do that you need a good idea what the total reset set of your search query because if there were only 43 hits, you don’t want to shot options to jump to page 9. The property that gives you the number you’re looking for is ResultTable.TotalResults. Now there’s something special about this guy: he doesn’t give you an exact number… he gives you an estimate. Why an estimate? Quite simply, with all the security trimming and other complex logic inherit to search algorithms, it’s just too expensive to get a specific number. Sites like Live.com can do this because they don’t have to concern themselves with the security trimming of hits.

But this is not all… there’s another property you should pay attention to: Query.TotalRowsExactMinimum. This property tells SharePoint this is the minimum number of hits to be included in the search. It’s used to generate the estimate of total results. Think of it like a hint to search… saying “you only have to work this hard on this query.” Most search implementations only show the next few paging options… they don’t show ALL the options. For instance, if you’re on page 5, your paging control may show the following:

«Previous« 2 3 4 5 6 7 8 »Next»

In this case, you don’t need for search to find ALL the results… you only need it to determine how many more page options you want to show to see if you’re going to show too many or too few. In the above example, you have an additional 3 pages of results you want to show. Continuing on this example, you have a result set of 15 and you have an additional 3 pages you want to show, the Query.TotalResultsExactMinimum property would be 45 as it already is going to factor into the equation the Query.StartRow property:

using (FullTextSqlQuery query = new FullTextSqlQuery(SPContext.Current.Site))

{

query.StartRow = 16;

query.RowLimit = 15;

// TotalRowsExactMinimum = [number of pages to show] * [page size]

query.TotalRowsExactMinimum = 45;

query.QueryText = "SELECT Rank, Title Url FROM Scope() WHERE FREETEXT(defaultproperties,'sales proposal') ORDER BY Rank Desc",

ResultTableCollection results = query.Execute();

ResultTable relevantResults = results[ResultType.RelevantResults];

// do work with the results

}

That’s all there really is to it! One parting word of advice: use the Query.TotalRowsExactMinimum property with care as the higher its set, the greater performance impact there will be on each search query executed.

A special shout out & thanks to Puneet Narula @ Microsoft for helping uncover this very helpful nugget of info.

Andrew Connell (blog)
Microsoft MVP

System Center Data Protection Manager & Search

enterprisesearch — Wed, 21 May 2008 20:44:44 GMT

Microsoft® System Center Data Protection Manager (DPM) is designed to help you take backups and restore data easily. For a Microsoft SharePoint® farm, DPM understands the objects within the farm and backs up the most relevant data with least amount of user intervention. DPM 2007 backs up:

1. Configuration database which stores most of the farm settings.

2. Administrator content database which stores the content of the central admin website.

3. Individual content DBs that store information about specific sites, their subsites, document libraries and documents.

Apart from this, DPM customers have highlighted a need to back up SharePoint Search service as well – both Windows SharePoint Services (WSS) Search and Microsoft Office SharePoint Server (MOSS) 2007 Search.

Backing up search is not a trivial problem as the Search service is continuously running background processes (like crawlers) on the SharePoint farms and other locations to index the files in the Search catalog. These programs keep updating the Search service data on continuous basis. Moreover, the search data is not restricted to one data source. It is spread across multiple data sources which may in turn be located across different servers. The following table shows the list of data sources used by the Search service:

Data Source	Data Type	WSS 3.0 Search	MOSS 2007 Search
Search database	SQL database	X	X
Index Files	Files	X	X
Shared Service Provider (SSP) database	SQL database		X

DPM uses the VSS’ snapshot technology that ensures consistency of data in a data source at a point in time. However, even this technology fails when data is being written across multiple data sources causing inconsistent backups resulting in corrupt data at the time of recoveries.

To ensure a good backup of SharePoint Search service, all background activity or processes updating the search data across its multiple data sources must be paused. The steps involved in backing up are:

1. Pause all the background activity and processes updating the search data sources

2. Create VSS snapshots of the data sources and send them to backup application (in this case DPM)

3. Resume the background activities and processes.

DPM has the infrastructure to enable such backups and ensure that the MOSS 2007 Search service is backed up in a consistent manner. The steps to backup and restore MOSS 2007 Search service using DPM 2007 can be found at WhitePaper: MOSS 2007 Search Backup for DPM 2007

Saurabh Bansal
Program Manager
Microsoft India.

Search training material available on TechNet!

enterprisesearch — Wed, 21 May 2008 03:09:30 GMT

We recently published ~17 hours of training videos for Enterprise Search on TechNet.

The 14 recorded presentations are based on training modules from a three-day, in-person training course ‘Implementing and Deploying an Enterprise Search solution’.

The presentations provide details about key Enterprise Search capabilities in Microsoft Office SharePoint Server 2007.

The content is recorded in English.

Michal Gideoni
Senior Product Manager
Microsoft Corp.

SQL Monitoring and I/O

enterprisesearch — Tue, 20 May 2008 06:45:15 GMT

Hello again, the interest in the existing posts is starting to climb. As always feel free to post comments or questions to any of the posts and I will either answer your questions directly or add them as topics for the next set of posts.

This week I wanted to call out some information around monitoring SQL from a Search perspective and provide some guidance around configuration of the SQL machine. There are many great articles out there that already document much of this information. So I am attempting to consolidate this information and provide you with my "reading list." I'll also summarize some of this data in the key areas you should focus on when working with the SQL box that Search is using.

Primary list of documents to read through, ordered in recommended reading order:

1. Planning and Monitoring SQL Server Storage for SharePoint - This is a good document to start with as it discusses many of the topics at a high-level. However, it does not discuss much about the SQL utilization that Search has.

2. SQL Storage Top 10 Best Practices - A great primer for issues to consider when building out hardware for the SQL box in a Search deployment.

3. Optimizing tempdb Performance - This is very applicable to Search and I strongly recommend you follow the guidance of this article. The tempdb is used in every end-user query executed. Plus, the crawler makes reasonable use of the tempdb. Making sure the tempdb is performant will directly impact the throughput and latencies of end-user queries.

a. Working with tempdb in SQL Server 2005 - A more detailed description of how and when the tempdb is used. Useful as supportive documentation as to why you should optimize the tempdb performance.

4. SQL Predeployment I/O Best Practices - This is a great article discussing the steps you should take to validate the I/O system of a SQL box prior to going into production.

5. Troubleshooting Performance Problems in SQL Server 2005 - This is a great paper discussing all of the various bottle-necks that SQL can have.

As hinted at above Search uses SQL in a very I/O intensive fashion. It is sensitive to I/O latencies on the TempDB and the Query and Crawl file groups. The basic recommendation from the SQL team is to keep your latency (Avg. Disk sec/Read and Avg. Disk sec/Write) less than 20 ms for OK performance. For Search I would strongly recommend:

10ms or less for TempDB. Both Search and content hosting make heavy use of Temp DB, at this scale point it is recommended that you split the content away from the SSP/Search.
10ms or less for the Query file group
20ms or less for the Crawl file group
20ms or less for the and database Log file

As you'll note below SearchBeta is currently unable to do this, we are in the process of obtaining new hardware (more disks) to rectify this. Knowing I had limited hardware going into this project I allocated more spindles to the TempDB and Query file-group than the other files. See SearchBeta Hardware definition for more details.

With our hardware we are close to meeting our crawl freshness goals of less than 24 hours for the high-value repositories. We see 24 hours fresh for the smaller sites and around 50 hours for the bigger ones. Our query latencies tend to be in the 2 second range for 95% of our measured queries.

IOPs, Throughput and Latencies:

Drive	IOPs Read (max)	IOPs Write (max)	Ratio Read/ Write	Throughput Read (bytes)	Throughput Write (bytes)	Latency Read (sec)	Latency Write (sec)
Search DB Logs	14.67	1,777.29	0.01	901,126	64,557,167	0.3060	0.8550
Temp DB	1,110.98	1,492.01	0.74	72,808,827	97,770,866	1.6870	3.5660
Query file group	3,507.26	1,631.96	2.15	148,370,386	126,034,214	3.4360	3.2140
Crawl file group	3,043.93	371.65	8.19	104,533,884	10,261,624	15.0840	15.8720

For comparison purposes I have included the current IOPs, throughput and latency numbers that I am getting on SearchBeta. These numbers are for comparison purposes and should be useful as a starting point for configuring your hardware. However, you should make sure that you test you pre-deployment environment to verify that you are within the recommended latency ranges. It also recommended that you baseline your production system and periodically spot check to verify that you have not deviated away from your baseline. Overtime you will see growth in the amount of content you are indexing and the volume of queries being executed. Having a baseline in place and a process to verify the live system latencies will allow you detect problems sooner.

I've dedicated most of this post to I/O intentionally as this will be the key bottle-neck that you will want to architect the system for. The white paper Planning and Monitoring SQL Server Storage for SharePoint discusses memory a little more. But in general; if you are deploying a larger scale (> 10 million indexed documents) MSS deployment then your SQL box should be OK with 16GB of RAM. However, if you are deploying a MOSS environment where you will be hosting People, Usage Analysis and other MOSS features you will want to start with 32GB.

That is all for this post. I know you are all waiting for information on SQL maintenance and information on how to create the Crawl and Query file group. These are the next items on my priority list and I hope to have the details out soon

-Thanks

Dan Blood
Senior Test Engineer
Microsoft Corp