Microsoft Enterprise Search Blog : Tools

The Search Developer Story in SharePoint 2010 - Query Interfaces

enterprisesearch — Fri, 20 Nov 2009 23:39:00 GMT

SharePoint 2010 includes a number of features that make the platform easier to use for developers. An improved Visual Studio integration, the addition of LINQ to the SharePoint platform, sandboxing for deployment, and the new developer dashboard are just a few examples of how developing and deploying SharePoint solutions have become much easier.

As a member of the enterprise search development team that has worked to bring FAST Search into SharePoint 2010, I can tell you that a lot has also been done to benefit developers of search-based solutions. SharePoint 2010 Search and the new FAST Search for SharePoint 2010 have been designed to share a common platform so that search developers can integrate with both SharePoint Search and FAST Search for SharePoint 2010 using the same query side interfaces. This means developers don’t have to learn new APIs or programming models, but can leverage the same object models, services and a common query language for both products.

SharePoint developers and architects implementing search-driven applications should understand the available integration options. Depending on requirements, tools, and preferences, one can choose from among several integration points, including a brand new object model in SharePoint 2010. Here’s a list of the different integration points with a brief description of each:

The Federation Object Model (OM)

This is a new search object model in SharePoint 2010. It provides a unified interface for querying against different locations (search providers), giving developers of search-driven Web Parts a way to implement end-user experiences that are independent of the underlying search engine. The object model also allows for combining and merging results from different search providers. Out-of-box Web Parts in SharePoint 2010 are based on this OM, and SharePoint 2010 ships with 3 different types of locations; SharePoint Search, FAST Search and OpenSearch. The Federation OM is also extensible, should you want or need to implement a custom search location outside of the supported types.

The Query Web Service

This is the integration point for applications outside of your SharePoint environment, such as standalone, non-web based applications, or Silverlight applications running in a browser. The Query Web Service is a SOAP based ASMX web service, and supports a number of operations, including:

Querying and getting search results

Getting query suggestions

Getting meta data, e.g. a list managed properties

The same schema is shared for SharePoint Search and FAST Search, and both products support the same operations. For querying, clients can easily switch the search provider by setting a ResultsProvider element in the request XML. A number of extensions are available for FAST Search, e.g. refinement results, advanced sorting using a formula, issuing queries using the FAST Query Language.

The Query RSS Feed

Certain scenarios, like simple mashups, may need only a simple search result list. The RSS feed is an alternative, lightweight integration point for supplying applications outside of SharePoint with a simple RSS result list. The Search Center - the default search front-end in SharePoint 2010 - includes a link to a query-based RSS feed. Switching the engine to the RSS format is done by simply setting a URL provider. Because of its intended simplicity, there are some limitations to what can be returned and customized in the query RSS feed. The object models or the web service integration scenarios are recommended for more advanced applications.

The Query Object Model

This is the lowest level object model, used by the Federation object model, the Query Web Service and the Query RSS feed. Both SharePoint Search and FAST Search support the KeywordQuery object in this object model. While the Federation OM returns XML (to Web Parts), the Query OM returns data types.

The Search Web Parts

Search Web Parts in SharePoint 2010 are common in SharePoint Search and FAST Search, and are now based on the common Federation OM. The Web Parts on a page communicate through a shared Query Manager, a central component of the Federation OM. This makes adding new Web Parts that interact with existing Web Parts simpler than before. For example, a new Tag Cloud Web Part for visualizing the query results can utilize the shared Query Manager for getting results. Developers will also be able to extend out-of-box Web Parts as they now are public in SharePoint 2010 (no longer sealed).

The Common Query Language

Both SharePoint Search and FAST Search support the Keyword Query Language syntax. This is the default query language for both products, and the end-user language supported from the Web Parts in the search centers (including the advanced search page).

FAST Search Extensions

FAST Search has a number of extensions beyond the standard SharePoint Search that are available on both the Federation and Query object models, and as well as on the query web service. Some examples are:

The FAST Query Language, which supports advanced query operators like XRANK for dynamic (query time) term weighting and ranking.
Deep refiners over the whole results set, and the possibility of adding refiners over any managed property
Advanced sorting using managed properties or a query-time sort formula.
Advanced duplicate trimming, with the ability to specify a custom property on which to base duplicate comparisons.
“Similar documents” matching.
FAST Search Admin Object Model for promoting documents or assigning visual best bets to query keywords/phrases.

Building powerful search applications is easier than ever in SharePoint 2010. FAST Search is now integrated into the SharePoint platform and developers of search-driven solutions and applications can leverage a common platform and common APIs for both SharePoint Search and FAST Search. This means applications can be built to support both search engines and then extended if and when desired to take advantage of the more advanced features available with FAST Search, such as dynamic ranking, flexible sort formulae, or deep refiners for insight into your full result set.

Arnt Schøning, Senior Development Engineer | Microsoft Enterprise Search Group

(o on Twitter as @aschoning)

Announcing: SharePoint Web Parts for FAST ESP

enterprisesearch — Fri, 20 Jun 2008 20:06:00 GMT

It’s been around 45 days since the acquisition of FAST Search and Transfer closed and we’re moving quickly to provide interoperability for Microsoft customers between FAST ESP and Microsoft SharePoint Server.

The first deliverables from this work are a set of FAST ESP Search Web Parts for quickly integrating results from FAST ESP into SharePoint Server 2007 and a FAST ESP Search site template.

Using these Web Parts and Site Template SharePoint administrators will be able to quickly and easily build FAST ESP-based search sites inside SharePoint 2007 by simply dropping in and configuring the appropriate components.

The Web Parts and Site Template are available as a free download (both compiled code and source code) from CodePlex at www.codeplex.com/espwebparts and are part of the Search Community Toolkit.

Some of the FAST ESP search capabilities that can be exposed within SharePoint Server 2007 using these Web Parts include:

• Search Box Web Part -- Search box for query term submission and includes “did you mean” functionality for query correction

• Result List Web Part -- Displays search results and supports sorting, pagination, and navigator-based filtering

• Navigator Web Part -- Displays dynamic navigators that profile search results across a set of pre-defined dimensions and allow users to refine the search through navigation clicks

• Breadcrumb Web Part -- Displays the search term(s) and list of navigators used to obtain the current result set

The FAST ESP Web parts are designed to be open and extensible, and we’re actively encouraging customers and partners to download them, customize them to align with their branding and extend them to fit their search and user experience requirements.

Expect the features, functionality and range of ESP Web Parts to grow through contributions from the search developer community as well as further contributions from the FAST & Microsoft Search Team!

FAST & Microsoft Search Teams.

Introducing Protocol Handler.NET

enterprisesearch — Thu, 05 Jun 2008 09:13:27 GMT

"Protocol Handler.Net is a set of .Net wrappers for the protocol handler interfaces that enable developers to create and deploy protocol handlers for SharePoint search and Search Server.

Developers can index data and documents from any system they can connect to.

Much of the complexity and time around the development of protocol handlers, such as COM interoperability, are reduced and hidden in the wrappers themselves letting developers just concentrate on code to connect to a content source and pull data.

Protocol Handler.Net makes it possible to develop protocol handlers in C# or VB.Net and simplifies the handling of security, metadata, streaming content, deployment and management just to name a few things. It also comes with a help system and sample project to further help developers."

Big thank you to to Chris Gomez from http://www.FastSharePoint.com for creating and sharing these tools and samples!

They are available now on CodePlex at http://www.codeplex.com/phdotnet and are part of the Search Community Toolkit

Introducing the ExportCrawlLog STSADM Command Extension

enterprisesearch — Tue, 27 May 2008 06:46:21 GMT

In versions of SharePoint prior to MOSS 2007 each time a crawl was executed, a new group of log messages were stored to the database. Also, the name of the log itself was changed in the documentation and the user interface. Formerly this log was known as the Gatherer Log, but it is now called the Crawl Log.

When troubleshooting problems with the crawl of a particular content source it was (and still is) sometimes useful to compare and contrast the messages logged between one crawl and the next. In MOSS 2007, the storage of the crawl log messages has been optimized/minimized such that only the most recent message for a given URL is stored in the database. As a consequence, the results from a prior crawl are overwritten by results from subsequent crawls. In other words, you can only ever see the most recent log message for a given URL.

This is where the STSADM command extension “ExportCrawlLog” comes in. The motivation for preparing this tool is to provide a way to make a “snapshot” of the Crawl Log information at a point in time to facilitate post-mortem analysis of crawl problems. As a bonus, in addition to extracting crawl log detail, it also provides some summary reporting features. The goal of the tool is to provide a means of gathering data by which you can track and manage the health of your index over time. For instance you could setup a scheduled task to run this command once a day and generate summary reports that can provide data for trend monitoring.

ExportCrawlLog uses only the published APIs of the SharePoint Object Model and must be run on the index server of your SharePoint Farm. ExportCrawlLog is available as source code on Codeplex at http://www.codeplex.com/ExportCrawlLog and is part of the Search Community Toolkit.

Please use the Discussion tracking and Issue tracking features of Codeplex to offer your feedback.

Larry Kuhn
Architect
Microsoft Consulting Services.

Search Relevance Tuning

enterprisesearch — Thu, 15 May 2008 03:37:51 GMT

Changing the advanced relevance settings for Search Server 2008 and SharePoint Server 2007 (such as the global ranking parameters, property weights and property length normalization) isn't something that you'll want to do without a lot of thought, planning and testing.

And in the vast majority of cases you'll never need to go near the settings, but in certain specialized deployment scenarios it might be something you want to consider.

This MSDN article and TechNet White Paper give you a very detailed review of the relevance knobs and wheels you can turn along with how to structure and plan your testing to see if you're achieving your desired end result.

To help you on your way, Christopher Even (http://www.sharepointsearch.com) has written and shared on CodePlex a small utility that allows you to change all of the advanced relevance settings quickly and easily.

The tool is here and is part of the Search Community Toolkit.

This tool comes with a relevance health warning: Read the MSDN and TechNet content before you start, use the tool with care, make a note of all of the values that you change before you change them and ensure you're consistent and diligent in your relevance testing after you make any changes!

There's nothing that the tool does that can't be done though the search object model, it just saves you having to write code to tune the various advanced parameters.

Richard Riley
Senior Technical Product Manager
Microsoft Corp

Crawling Novell Netware with SharePoint Server 2007 and Search Server 2008

enterprisesearch — Tue, 13 May 2008 01:07:03 GMT

Search in SharePoint Server 2007 and Search Server 2008 can crawl a variety of content sources. For several of the “out of the box” content sources security information in the form of ACLs are collected as part of the indexing process. At query time, search results are trimmed based on the identity of the user submitting the query, and the ACLs collected from crawling.

In certain scenarios, the built-in security trimming is not sufficient for your requirements or the indexing connector isn’t able to collect ACLs at crawl time - in this scenario you'll need to implement a Custom Security Trimmer. For example, content in Novell Netware file shares can be crawled by SharePoint Server 2007 & Search Server 2008(Using the built-in Fileshare Indexing Connector along with Windows Services for Netware) but the indexing connector doesn’t know how to collect ACLs from Netware.

For security trimmed results, that scenario will require a custom security trimmer (CST) to be developed. We recently released a Novell Netware CST on CodePlex as part of the Search Community Toolkit. This can be installed and configured in less than 30 minutes and has been tested against Netware 6.5 although it should work with earlier versions too. It's compatible with SharePoint Server 2007, Search Server 2008, and Search Server 2008 Express.

It includes documentation on how to configure a content source to crawl Netware and how to install the CST. We’ll post the source code for the CST soon once we’ve finished some final testing and the code will be commented and include a doc that provides an overview of the structure of the code and explains where you might want to make your own modifications. This will be great example of how to build your own CST if you’re looking for a good sample to start from.

The initial version was released with an open source license at http://www.codeplex.com/sctcstn and includes the following features:

Netware Custom Security Trimmer Binaries
Documentation Providing Click Through Guidance

For Crawling Netware Content
For Installing the Binaries
For Security Trimming Netware Results

Semi-Automated Installation In Under 30 minutes
Forms Authentication Support
64 bit Platform Support

Mitch Prince
Architect
Microsoft

John Kelly
Consultant
Microsoft

Search Community Toolkit

enterprisesearch — Wed, 12 Mar 2008 23:15:33 GMT

Last week at the SharePoint Conference we announced the Search Community Toolkit which is a set of tools, utilities and samples available from CodePlex to extend and enhance the search features in SharePoint Server 2007, SharePoint Server 2007 for Search and the recently announced Search Server and Search Server Express.

Some of the tools and utilities have been around for a while and have proved very popular such as Faceted Search and others are brand new. Two new additions are;

Search ASP.NET Server Controls - These 2 controls replicate the Search Box and Core Results Web Parts but as ASP.NET Server Controls bound to the search web service. Using these controls on an ASP.NET web site along with Search Search Express will give you powerful .com site search for FREE! As the search web service is identical between SharePoint Server, SharePoint Server for Search and Search Server these controls are compatible with all 3 products and product versions.

Search Center Branding Tool - This tool installs as a SharePoint Feature and allows you to very easily switch off the SharePoint chrome on the Search Center. The tool is primarily focused on Search Server Express and helps you easily brand a standalone instance of Search Server or just remove the default SharePoint look and feel with a single click.

If you want to get involved, have feedback or ideas for other tools then please feel free to get in touch!

Richard Riley
Senior Technical Product Manager
Microsoft Corp