Microsoft Enterprise Search Blog : Query

The Search Developer Story in SharePoint 2010 - Query Interfaces

enterprisesearch — Fri, 20 Nov 2009 23:39:00 GMT

SharePoint 2010 includes a number of features that make the platform easier to use for developers. An improved Visual Studio integration, the addition of LINQ to the SharePoint platform, sandboxing for deployment, and the new developer dashboard are just a few examples of how developing and deploying SharePoint solutions have become much easier.

As a member of the enterprise search development team that has worked to bring FAST Search into SharePoint 2010, I can tell you that a lot has also been done to benefit developers of search-based solutions. SharePoint 2010 Search and the new FAST Search for SharePoint 2010 have been designed to share a common platform so that search developers can integrate with both SharePoint Search and FAST Search for SharePoint 2010 using the same query side interfaces. This means developers don’t have to learn new APIs or programming models, but can leverage the same object models, services and a common query language for both products.

SharePoint developers and architects implementing search-driven applications should understand the available integration options. Depending on requirements, tools, and preferences, one can choose from among several integration points, including a brand new object model in SharePoint 2010. Here’s a list of the different integration points with a brief description of each:

The Federation Object Model (OM)

This is a new search object model in SharePoint 2010. It provides a unified interface for querying against different locations (search providers), giving developers of search-driven Web Parts a way to implement end-user experiences that are independent of the underlying search engine. The object model also allows for combining and merging results from different search providers. Out-of-box Web Parts in SharePoint 2010 are based on this OM, and SharePoint 2010 ships with 3 different types of locations; SharePoint Search, FAST Search and OpenSearch. The Federation OM is also extensible, should you want or need to implement a custom search location outside of the supported types.

The Query Web Service

This is the integration point for applications outside of your SharePoint environment, such as standalone, non-web based applications, or Silverlight applications running in a browser. The Query Web Service is a SOAP based ASMX web service, and supports a number of operations, including:

Querying and getting search results

Getting query suggestions

Getting meta data, e.g. a list managed properties

The same schema is shared for SharePoint Search and FAST Search, and both products support the same operations. For querying, clients can easily switch the search provider by setting a ResultsProvider element in the request XML. A number of extensions are available for FAST Search, e.g. refinement results, advanced sorting using a formula, issuing queries using the FAST Query Language.

The Query RSS Feed

Certain scenarios, like simple mashups, may need only a simple search result list. The RSS feed is an alternative, lightweight integration point for supplying applications outside of SharePoint with a simple RSS result list. The Search Center - the default search front-end in SharePoint 2010 - includes a link to a query-based RSS feed. Switching the engine to the RSS format is done by simply setting a URL provider. Because of its intended simplicity, there are some limitations to what can be returned and customized in the query RSS feed. The object models or the web service integration scenarios are recommended for more advanced applications.

The Query Object Model

This is the lowest level object model, used by the Federation object model, the Query Web Service and the Query RSS feed. Both SharePoint Search and FAST Search support the KeywordQuery object in this object model. While the Federation OM returns XML (to Web Parts), the Query OM returns data types.

The Search Web Parts

Search Web Parts in SharePoint 2010 are common in SharePoint Search and FAST Search, and are now based on the common Federation OM. The Web Parts on a page communicate through a shared Query Manager, a central component of the Federation OM. This makes adding new Web Parts that interact with existing Web Parts simpler than before. For example, a new Tag Cloud Web Part for visualizing the query results can utilize the shared Query Manager for getting results. Developers will also be able to extend out-of-box Web Parts as they now are public in SharePoint 2010 (no longer sealed).

The Common Query Language

Both SharePoint Search and FAST Search support the Keyword Query Language syntax. This is the default query language for both products, and the end-user language supported from the Web Parts in the search centers (including the advanced search page).

FAST Search Extensions

FAST Search has a number of extensions beyond the standard SharePoint Search that are available on both the Federation and Query object models, and as well as on the query web service. Some examples are:

The FAST Query Language, which supports advanced query operators like XRANK for dynamic (query time) term weighting and ranking.
Deep refiners over the whole results set, and the possibility of adding refiners over any managed property
Advanced sorting using managed properties or a query-time sort formula.
Advanced duplicate trimming, with the ability to specify a custom property on which to base duplicate comparisons.
“Similar documents” matching.
FAST Search Admin Object Model for promoting documents or assigning visual best bets to query keywords/phrases.

Building powerful search applications is easier than ever in SharePoint 2010. FAST Search is now integrated into the SharePoint platform and developers of search-driven solutions and applications can leverage a common platform and common APIs for both SharePoint Search and FAST Search. This means applications can be built to support both search engines and then extended if and when desired to take advantage of the more advanced features available with FAST Search, such as dynamic ranking, flexible sort formulae, or deep refiners for insight into your full result set.

Arnt Schøning, Senior Development Engineer | Microsoft Enterprise Search Group

(o on Twitter as @aschoning)

How to: Mine the ULS logs for query latency

enterprisesearch — Wed, 03 Sep 2008 02:38:00 GMT

Tracking query latencies can be made easier through the use of the products ULS logs. Below you will find information on how to enable the specific ULS traces as well as information for how to parse the logs. The primary usage of this information is to monitor the ongoing health of your system. It is one tool in the toolbox to make sure that the system is running in a viable state. It is also necessary when you are making small changes to your environment so you can measure the benefits or detriments of the changes made. Another key usage of the query latency ULS logs is the ability to where the larger portions of time is being spent in the query. For example you can see the time spent in SQL improve after doing index defrags.

ULS logging

Making changes to ULS log settings can impact performance and cause more disk space to be consumed when. However, the category and level changes mentioned below are what SearchBeta is running with and the cost of this is negligible given the benefit it provides. Just make sure your logs files are not on a drive that is tight on disk space.

You will need to change the following ULS settings to get the events that we need traced.

From "Central Admin.Operations.Diagnostic Logging" set the following;
Category: "MS Search Query Processor"
Least critical event to report to the trace log: "High"

LogParser

There are a number of interesting traces that you get with the above setting. To really look at this data you will need to use some kind of log parsing utility to strip out the interesting traces and perform some additional post processing. I recommend that you use logparser.exe to do this parsing. Below I give examples of Log Parser queries to get at the data. Additionally you should provide the following input parameters to logparser.exe since the ULS log files are Unicode, tab separated text files.

-i:TSV -iCodepage:-1 -fixedSep:ON

Traces

With the above ULS trace settings you will get the following messages in the log (location of these log files can be found in the above UI for changing the logging level):

Completed query execution with timings: v1 v2 v3 v4 v5 v6
- The 5 numbers v1,v2,v3,v4,v5, and v6 are time measurements in milliseconds
  - v6 = Cumulated time spent in various calls to SQL except the property fetching
  - v5 = Time spent waiting for the full-text query results from the query server (TimeSpentInIndex)
  - v4 = Latency of the query measured after the joining of index results with the SQL part of the query. This includes v5 and the time spent in SQL for resolving the SQL part of advanced queries (e.g. queries sorted by date or queries including property based restrictions like AND size > 1000).
  - v4-v5 = Join tim
  - v3 = Latency of the query measured after security trimming. It includes V4 plus retrieval of descriptors form SQL and access check.
  - v3-v4 = Security Trimming tim
  - v2 = Latency of the query measured after the duplicate detection.
  - v3-v2 = Duplicate detection tim
  - v1 = Total time spent in QP. (TotalQPTime)
  - v1 -v2= Time spent retrieving properties and hit highlighting . (FetchTime)
Join retry v1 v2 v3
- Retry caused because there were not enough results from SQL that matched the results returned from the full-text index.
Security trimming retry v1 v2 v3
- Caused by the user executing a query the returns a number of results that they do not have permission to read. The query is retried until the enough results are available to display the first page of results.
Near duplicate removal retry v1 v2 v3
- There were so many virtually identical documents that were trimmed out that the query processor did not have an adequate number of documents to display
- The 3 numbers v1,v2 and v3 are counts of documents. If you see one of these messages in the log it means that the query processor was unable to satisfy the requested number of results on the first attempt and had to execute the SQL portion of the query a second++ time with a larger number of requested results. The numbers here are not excessively useful and most of the analysis you will do is around the existence of this trace. This and the relative frequency of each of the retries allows you to determine why so much time is being spent in a given phase of the query.
  - v1 is the current upper bound on the number of documents to work with (this will go up on subsequent retries)
  - v2 is the number of documents before the operation that caused the retry
  - v3 is the number of documents after the operation that caused the retry.

Where is all of the time being spent for the queries executed in the system?

The answer to this question is primarily within the "Completed query execution…" trace. The number of retries help explain why the time spent in any one location is so high. Given all of the timing information that you can get from a single query and the fact that this data is available for each and every query executed, the problem becomes more of an exercise in figuring out how to store the data and provide a mechanism to summarize or chart it. Without doing this there is just too much data to try and interpret. The solution we have on SearchBeta is to collect the data on a regular basis (hourly) and import it into a SQL reporting server that is segregated from the SQL machine hosting the Search farm.

Once the data is in SQL we have created a number of Excel spreadsheets that query the data directly from SQL and chart it using Excel Pivot Tables/Charts. We have also gone further to provide a set of dashboards within a MOSS system that use Excel Server to provide up to date reports on the health of the system that are available for anyone to look at.

Once you have the basics of this system set-up there are a multitude of other reports and health monitoring that are possible; from collecting performance counters to mining IIS logs. The IIS logs provide a key piece of information about query latencies that is missing from the ULS trace. Primarily answering the question of how much additional time is spent rendering the UI.

A sample of one of the charts that we are able to produce with the ULS log data is below:

The log parser query that we use to mine the ULS logs is below. Note there are number of output options for LogParser, I am using a simple CSV file below. But you can also import the data directly into SQL.

*remember the numbers in the log are in milliseconds, the query below translates the time into seconds.

Select  Timestamp
      , TO_INT(Extract_token(Message,7, ' ')) as TotalQPTime
      , TO_INT(Extract_token(Message,8, ' ')) as v2
      , TO_INT(Extract_token(Message,9, ' ')) as v3
      , TO_INT(Extract_token(Message,10, ' ')) as v4
      , TO_INT(Extract_token(Message,11, ' ')) as TimeSpentInIndex
      , TO_INT(Extract_token(Message,12, ' ')) as v6
      , SUB(v4, TimeSpentInIndex) as JoinTime
      , SUB(v3, v4) as SecurityTrimmingTime
      , CASE v2
            WHEN 0 THEN 0 
            ELSE SUB(v2, v3) 
        End as DuplicateDetectionTime
      , SUB(TotalQPTime, v2) as FetchTime
INTO QTiming
FROM \\%wfeHost%\ULSlogs\%wfeHost%*.log
WHERE Category = 'MS Search Query Processor' 
      AND Message LIKE '%Completed query execution with timings:%'

*FYI -- Prior to the MSS release and Infrastructure Update updating MOSS with the MSS changes, the first two "tokens" (QueryID: XXX.) at the beginning of the trace did not exist. So you will need to subtract 2 from the second parameter of each "Extract_token" predicate in the above SQL command.

What is the percentage of retries that the system has?

To get an idea for how many "retries" are occurring you need to correlate the number of retries with the number of queries executed and calculate a % of total retry values for each type of retry. The timing data above does include time spent in a retry.

Log Parser queries:

Total number of queries executed

SELECT count (Message) 
FROM *.log 
WHERE Category = 'MS Search Query Processor' and Message 
like '%Completed query execution%'

Total number of retries due to Security trimming

SELECT count (Message) 
FROM *.log 
WHERE Category = 'MS Search Query Processor' and Message 
like '%Security trimming retry%'

Total number of retries due to Join retries

SELECT count (Message) 
FROM *.log 
WHERE Category = 'MS Search Query Processor' and Message 
like '%Join retry%'

Total number of retries due to Duplicate Removal

SELECT count (Message) 
FROM *.log 
WHERE Category = 'MS Search Query Processor' and Message 
like '%Near duplicate removal retry%'

Thank you for your time and as always I welcome any feedback or questions

Dan Blood
Senior Test Engineer
Microsoft Corp

Understanding Total Hits & Paging in the MOSS 2007 Search API

enterprisesearch — Thu, 22 May 2008 21:45:06 GMT

One of the more discussed topics I’ve seen (and struggled with myself) is around the concept of obtaining the total number of hits in a search results when working with the MOSS API. For instance, when I search for “sales forecast” in my SharePoint site, I want to not only see a set of paged results, 10 hits per page, but also see that my search found 127 matches. Those of you who’ve worked with the SharePoint Search Web Parts know this is a piece of cake using the Search Core Results, Search Paging and Search Statistics Web Parts.

But what if you need to roll your own solution? How can you get the same data out of your search query using the MOSS 2007 Search API? At first this can be a bit tricky but this post will hopefully show you how to knock it down to being a trivial task.

Executing a search query against the SharePoint API has you working with two objects that implement the abstract class Microsoft.Office.Server.Search.Query.Query: Microsoft.Office.Server.Search.Query.KeywordQuery and Microsoft.Office.Server.Search.Query.FullTextSqlQuery. The former KeywordQuery is useful for simple queries whereas the latter FullTextSqlQuery is much more powerful. Both implement the Execute() method which executes the defined query and returns back a collection of results as type Microsoft.Office.Server.Search.Query.ResultTableCollection. Using this object, you can get the specific results you are interested in. For instance to get the relevant results use the following to get an instance of a specific Microsoft.Office.Server.Search.Query.ResultTable:

using (FullTextSqlQuery query = new FullTextSqlQuery(SPContext.Current.Site))

{

query.QueryText = "SELECT Rank, Title Url FROM Scope() WHERE FREETEXT(defaultproperties,'sales proposal') ORDER BY Rank Desc",

ResultTableCollection results = query.Execute();

ResultTable relevantResults = results[ResultType.RelevantResults];

// do work with the results

}

Simple enough, but the project requires much more than that as usual. What we need to do is page the results to show only 15 items per page. No problem… let’s just modify that query a bit to set the Query.StartRow & Query.RowLimit properties of the query to say what page we’re on and tell SharePoint how many results we want to get back. Take for instance if we’re on page 2 of the results… we want to start with the 16^th hit as 1-15 were on page 1:

using (FullTextSqlQuery query = new FullTextSqlQuery(SPContext.Current.Site))

{

query.StartRow = 16;

query.RowLimit = 15;

query.QueryText = "SELECT Rank, Title Url FROM Scope() WHERE FREETEXT(defaultproperties,'sales proposal') ORDER BY Rank Desc",

ResultTableCollection results = query.Execute();

ResultTable relevantResults = results[ResultType.RelevantResults];

// do work with the results

}

Again… pretty straight forward. Now is where it gets a bit tricky. You need to show links to provide paging… but in order to do that you need a good idea what the total reset set of your search query because if there were only 43 hits, you don’t want to shot options to jump to page 9. The property that gives you the number you’re looking for is ResultTable.TotalResults. Now there’s something special about this guy: he doesn’t give you an exact number… he gives you an estimate. Why an estimate? Quite simply, with all the security trimming and other complex logic inherit to search algorithms, it’s just too expensive to get a specific number. Sites like Live.com can do this because they don’t have to concern themselves with the security trimming of hits.

But this is not all… there’s another property you should pay attention to: Query.TotalRowsExactMinimum. This property tells SharePoint this is the minimum number of hits to be included in the search. It’s used to generate the estimate of total results. Think of it like a hint to search… saying “you only have to work this hard on this query.” Most search implementations only show the next few paging options… they don’t show ALL the options. For instance, if you’re on page 5, your paging control may show the following:

«Previous« 2 3 4 5 6 7 8 »Next»

In this case, you don’t need for search to find ALL the results… you only need it to determine how many more page options you want to show to see if you’re going to show too many or too few. In the above example, you have an additional 3 pages of results you want to show. Continuing on this example, you have a result set of 15 and you have an additional 3 pages you want to show, the Query.TotalResultsExactMinimum property would be 45 as it already is going to factor into the equation the Query.StartRow property:

using (FullTextSqlQuery query = new FullTextSqlQuery(SPContext.Current.Site))

{

query.StartRow = 16;

query.RowLimit = 15;

// TotalRowsExactMinimum = [number of pages to show] * [page size]

query.TotalRowsExactMinimum = 45;

query.QueryText = "SELECT Rank, Title Url FROM Scope() WHERE FREETEXT(defaultproperties,'sales proposal') ORDER BY Rank Desc",

ResultTableCollection results = query.Execute();

ResultTable relevantResults = results[ResultType.RelevantResults];

// do work with the results

}

That’s all there really is to it! One parting word of advice: use the Query.TotalRowsExactMinimum property with care as the higher its set, the greater performance impact there will be on each search query executed.

A special shout out & thanks to Puneet Narula @ Microsoft for helping uncover this very helpful nugget of info.

Andrew Connell (blog)
Microsoft MVP

Search Relevance Tuning

enterprisesearch — Thu, 15 May 2008 03:37:51 GMT

Changing the advanced relevance settings for Search Server 2008 and SharePoint Server 2007 (such as the global ranking parameters, property weights and property length normalization) isn't something that you'll want to do without a lot of thought, planning and testing.

And in the vast majority of cases you'll never need to go near the settings, but in certain specialized deployment scenarios it might be something you want to consider.

This MSDN article and TechNet White Paper give you a very detailed review of the relevance knobs and wheels you can turn along with how to structure and plan your testing to see if you're achieving your desired end result.

To help you on your way, Christopher Even (http://www.sharepointsearch.com) has written and shared on CodePlex a small utility that allows you to change all of the advanced relevance settings quickly and easily.

The tool is here and is part of the Search Community Toolkit.

This tool comes with a relevance health warning: Read the MSDN and TechNet content before you start, use the tool with care, make a note of all of the values that you change before you change them and ensure you're consistent and diligent in your relevance testing after you make any changes!

There's nothing that the tool does that can't be done though the search object model, it just saves you having to write code to tune the various advanced parameters.

Richard Riley
Senior Technical Product Manager
Microsoft Corp

Add Virtual Earth Interactive Maps to your Search Server Site

enterprisesearch — Wed, 02 Apr 2008 00:40:24 GMT

Are you trying to think of ways to make your Search Server site more interactive and graphical? You can use Federation as a way to enhance the functionality of your search result pages.

The Federated Search Web Part makes it possible to display more than results from OpenSearch (1.0/1.1) sites on your search results page. The Search Server 2008 SDK explains how to include results from SQL Server database queries and search sites that do not expose XML feeds (such as Atom or RSS). In both scenarios, you do this by means of a "connector," a light-weight interface that sends queries to a given location or database, places the results into a structured XML document, and sends that XML to a Federated Search Web Part.

This sample demonstrates how you can extend the connector concept to Web service requests other than basic search queries. It shows how to use a connector to pass an address string to Microsoft's MapPoint Web service in order to obtain the latitude and longitude coordinates for that address. Once you have those coordinates, displaying a Microsoft Virtual Earth map requires only the addition of some Javascript to your Federated Location definition file. See this Codeplex project for the sample code (along with a sample Federated location definition file) and an explanation of how to implement it on your own site.

This sample is part of the Search Community Toolkit

Jim Crowley
Programming Writer
Microsoft Corp

Helper Classes for the SharePoint Server 2007 Search Query Web Service

enterprisesearch — Wed, 03 Oct 2007 05:25:00 GMT

Sometimes, things just kind of grab hold of you, don't they? A corridor conversation leads to quickly popping into a meeting to give a bit of advice, and this leads to suddenly being given actions. And then, the sense of responsibility kicks in and you find yourself going the extra mile and coding something up to help make something easy.

It happened that way when I bumped into a colleague who was just going into a meeting. When the conversation turned to what he was doing, he mentioned that he was going into a meeting to discuss an internal project. It was nothing fancy or super-secret; in fact, it was a problem that I've been helping businesses with for the last few years. We have a series of community sites that are maintained by our technical team leads and subject matter experts. One of our senior managers wanted a single mechanism by which everyone could search all this content as well as publishing information that was relevant to the whole group. I quickly gave my standard "information portal coupled with enterprise search" speech and found myself invited to the meeting to repeat it.

So, I found myself helping out. One of the neat ideas was to use the search capability as an internal research and training exercise so that folks could play with developing a simple Silverlight application. No problem, I thought. This is going to be easy for me as all I have to do is set up the site, configure the search to crawl the team site, and the Silverlight developers can invoke the search web service. I could get the job done with minimum effort and maximum kudos.

Then the sense of responsibility kicked in. As I looked at the search web service, I realised that it would be really helpful if I gave the Silverlight developers a set of helper classes. These helper classes would make it easier for them to submit a query without having to worry about building XML query documents and invoking web services. After a few trials and errors, I ended up with three main classes: QueryRequest, QueryResponse, and MossQuery. The QueryRequest class is used to define the query that will be fired in, MossQuery actually invokes the Search web service and it massages the result to build a QueryResponse.

And then I thought, "Hey, I bet other folks could make use of these classes …" So I wrote an MSDN article that walked through the code and explained how it all worked. I popped the project up on Codeplex and was then invited to write an entry to announce it here.

So, let me know how it goes, and if the code is useful.

And if I bump into another colleague, perhaps there'll be another little coding project for me somewhere else ...

Callum Shillan
Solution Architect

Microsoft Consulting Services UK