Microsoft Enterprise Search Blog : Search Community Toolkit

Announcing Faceted Search v2.5

enterprisesearch — Wed, 13 Aug 2008 00:50:22 GMT

Starting Faceted Search 2.5, the solution relies on Microsoft Enterprise Library to address common software requirements in caching, logging, exception handling, policy injection etc., etc. More importantly, the 2.5 is a ground breaking release that is setting new targets for the Faceted Search. So, what’s new?

New Features

1. Caching – dramatically improves performance and decreases the load on the search engine

The solution uses 2 mechanisms for manageable cache: quick and long. I built the caching logic on assumption that user knows what he/she is looking for. The Search Facets web part will cache original result set and use it for the search refinement, paging and other postbacks. If the initial result set doesn’t provide full coverage of the search, the smart 2^nd thread will run against real-time data providing adjustment to the cached match.

2. Synchronization with Core Search Results web part

The MOSS search is adjusted by several parameters that designer can set for the Core Search Results web part itself. These include remove duplicates, enable trimming, permit noise words. When you drop the Search Facets web part to the search results page, it will find the Core Search Results, read its parameters and sync the search query parameters to exactly match ones used by the Core.

3. Support for advanced search

It was the most wanted feature since Faceted Search 1.0. With 2.5, the Facets are rendered for advanced search although do not extend yet to ranges. The functionality is accomplished by extending SearchQuery structure to accommodate POST requests and sync back to GET query.

4. Match of search counters

This release introduced an updated search syntax that is design to provide matching counters to the core search. In fact, the new search query is using both KeywordQuery and FullTextQuery through the use of generics.

public class GenericQuery<T> : IDisposable where T : Query
{
    private EventHandler _customLogic;

    public ResultTableCollection Execute(EventArgs args)
    {
        _customLogic(_query, args);
        return _query.Execute();
    }

    ...
}

Additionally, the WHERE clause of the search query was modified to provide closer match to the Core counter.

5. Introducing Parent-Child relationships

By design, the facets can support only 2 levels. This release extended the Facets schema to allow management of the nested layers. That eases the pain of displaying complex hierarchies such as geography, or org chart etc. Parent-Child relationship can be set by facet name and facet value, or just by facet name.

<Column Name="BDCCity" DisplayName="City" ParentName="BDCState" />
<Column Name="BDCState" DisplayName="State" >
  <Mappings>
    <Mapping Match="Alberta"  ParentName="BDCCountry" ParentValue="Canada"/>    
    <Mapping Match="Manitoba" ParentName="BDCCountry" ParentValue="Canada" />
    <Mapping Match="Ontario"  ParentName="BDCCountry" ParentValue="Canada"/>
    <Mapping Match="Quebec"   ParentName="BDCCountry" ParentValue="Canada"/>
  </Mappings>
</Column>

In the configuration above, the City facets will display only after the user chose the State. The State itself will match the country of origin.

6. Extending search to logical “OR” queries

Original facets always represent “AND” queries. That implies ability to narrow the search results by adding extra criteria. In this release I prototyped the way to expand the search by adding additional matches to the criteris. This in fact resulted in rewamped the Bread Crumbs UI. Proviuded now out-of-the-box support for languages is a good example of how “OR” queries empower the search.

7. Simplified web part properties

The 2.5 release is friendly to modifications of the web part properties. I have all properties classified and broken down to groups for each of the web parts.

8. Other

There are lots and lots of numerous fixes and enhancements, including improved security validation, code refactoring, extending facet sorting, support of quoted search and duplicates etc., etc.

What’s next

It’s my privilege to say that we have a team now that helps to shape new releases and brainstorm the furutre of the Faceted Search. In present we are looking at AJAX and SilverLight and hopefully you’ll start seeing more and more power of Facets in the near future.

Leonid Lyublinski
Senior Consultant
Microsoft Consultancy Services

Announcing: SharePoint Web Parts for FAST ESP

enterprisesearch — Fri, 20 Jun 2008 20:06:00 GMT

It’s been around 45 days since the acquisition of FAST Search and Transfer closed and we’re moving quickly to provide interoperability for Microsoft customers between FAST ESP and Microsoft SharePoint Server.

The first deliverables from this work are a set of FAST ESP Search Web Parts for quickly integrating results from FAST ESP into SharePoint Server 2007 and a FAST ESP Search site template.

Using these Web Parts and Site Template SharePoint administrators will be able to quickly and easily build FAST ESP-based search sites inside SharePoint 2007 by simply dropping in and configuring the appropriate components.

The Web Parts and Site Template are available as a free download (both compiled code and source code) from CodePlex at www.codeplex.com/espwebparts and are part of the Search Community Toolkit.

Some of the FAST ESP search capabilities that can be exposed within SharePoint Server 2007 using these Web Parts include:

• Search Box Web Part -- Search box for query term submission and includes “did you mean” functionality for query correction

• Result List Web Part -- Displays search results and supports sorting, pagination, and navigator-based filtering

• Navigator Web Part -- Displays dynamic navigators that profile search results across a set of pre-defined dimensions and allow users to refine the search through navigation clicks

• Breadcrumb Web Part -- Displays the search term(s) and list of navigators used to obtain the current result set

The FAST ESP Web parts are designed to be open and extensible, and we’re actively encouraging customers and partners to download them, customize them to align with their branding and extend them to fit their search and user experience requirements.

Expect the features, functionality and range of ESP Web Parts to grow through contributions from the search developer community as well as further contributions from the FAST & Microsoft Search Team!

FAST & Microsoft Search Teams.

Introducing Protocol Handler.NET

enterprisesearch — Thu, 05 Jun 2008 09:13:27 GMT

"Protocol Handler.Net is a set of .Net wrappers for the protocol handler interfaces that enable developers to create and deploy protocol handlers for SharePoint search and Search Server.

Developers can index data and documents from any system they can connect to.

Much of the complexity and time around the development of protocol handlers, such as COM interoperability, are reduced and hidden in the wrappers themselves letting developers just concentrate on code to connect to a content source and pull data.

Protocol Handler.Net makes it possible to develop protocol handlers in C# or VB.Net and simplifies the handling of security, metadata, streaming content, deployment and management just to name a few things. It also comes with a help system and sample project to further help developers."

Big thank you to to Chris Gomez from http://www.FastSharePoint.com for creating and sharing these tools and samples!

They are available now on CodePlex at http://www.codeplex.com/phdotnet and are part of the Search Community Toolkit

Introducing the ExportCrawlLog STSADM Command Extension

enterprisesearch — Tue, 27 May 2008 06:46:21 GMT

In versions of SharePoint prior to MOSS 2007 each time a crawl was executed, a new group of log messages were stored to the database. Also, the name of the log itself was changed in the documentation and the user interface. Formerly this log was known as the Gatherer Log, but it is now called the Crawl Log.

When troubleshooting problems with the crawl of a particular content source it was (and still is) sometimes useful to compare and contrast the messages logged between one crawl and the next. In MOSS 2007, the storage of the crawl log messages has been optimized/minimized such that only the most recent message for a given URL is stored in the database. As a consequence, the results from a prior crawl are overwritten by results from subsequent crawls. In other words, you can only ever see the most recent log message for a given URL.

This is where the STSADM command extension “ExportCrawlLog” comes in. The motivation for preparing this tool is to provide a way to make a “snapshot” of the Crawl Log information at a point in time to facilitate post-mortem analysis of crawl problems. As a bonus, in addition to extracting crawl log detail, it also provides some summary reporting features. The goal of the tool is to provide a means of gathering data by which you can track and manage the health of your index over time. For instance you could setup a scheduled task to run this command once a day and generate summary reports that can provide data for trend monitoring.

ExportCrawlLog uses only the published APIs of the SharePoint Object Model and must be run on the index server of your SharePoint Farm. ExportCrawlLog is available as source code on Codeplex at http://www.codeplex.com/ExportCrawlLog and is part of the Search Community Toolkit.

Please use the Discussion tracking and Issue tracking features of Codeplex to offer your feedback.

Larry Kuhn
Architect
Microsoft Consulting Services.

Crawling Novell Netware with SharePoint Server 2007 and Search Server 2008

enterprisesearch — Tue, 13 May 2008 01:07:03 GMT

Search in SharePoint Server 2007 and Search Server 2008 can crawl a variety of content sources. For several of the “out of the box” content sources security information in the form of ACLs are collected as part of the indexing process. At query time, search results are trimmed based on the identity of the user submitting the query, and the ACLs collected from crawling.

In certain scenarios, the built-in security trimming is not sufficient for your requirements or the indexing connector isn’t able to collect ACLs at crawl time - in this scenario you'll need to implement a Custom Security Trimmer. For example, content in Novell Netware file shares can be crawled by SharePoint Server 2007 & Search Server 2008(Using the built-in Fileshare Indexing Connector along with Windows Services for Netware) but the indexing connector doesn’t know how to collect ACLs from Netware.

For security trimmed results, that scenario will require a custom security trimmer (CST) to be developed. We recently released a Novell Netware CST on CodePlex as part of the Search Community Toolkit. This can be installed and configured in less than 30 minutes and has been tested against Netware 6.5 although it should work with earlier versions too. It's compatible with SharePoint Server 2007, Search Server 2008, and Search Server 2008 Express.

It includes documentation on how to configure a content source to crawl Netware and how to install the CST. We’ll post the source code for the CST soon once we’ve finished some final testing and the code will be commented and include a doc that provides an overview of the structure of the code and explains where you might want to make your own modifications. This will be great example of how to build your own CST if you’re looking for a good sample to start from.

The initial version was released with an open source license at http://www.codeplex.com/sctcstn and includes the following features:

Netware Custom Security Trimmer Binaries
Documentation Providing Click Through Guidance

For Crawling Netware Content
For Installing the Binaries
For Security Trimming Netware Results

Semi-Automated Installation In Under 30 minutes
Forms Authentication Support
64 bit Platform Support

Mitch Prince
Architect
Microsoft

John Kelly
Consultant
Microsoft