Open Source Faceted Search for MOSS 2007 and Microsoft Search Server 2008 - Part 2 of 2
Background
Hi, my name is Leonid Lyublinski, a consultant in Microsoft Consulting Services, and this is part 2 of the blog series that was started about a month ago with Neil Hodgkinson's post. He and I had a good time with our session at the Office Developer Conference 2008 in San Jose a few weeks ago. If you attended our session, thanks! We really appreciated your very positive evaluation scores and comments.
As a reminder, a good faceted search solution involves exposing the facets in dynamic taxonomies so that the user can see all of the refinement options at any time. The user can easily switch between a search based approach vs. metadata browsing, using a familiar terminology while recognizing the organization and vocabulary of the data.
Key features for metadata search include:
- Displaying aspects of the current results set in multiple categorization schemes.
- Showing only categories that have a result set, no dead-ends (links leading to empty lists).
- Displaying a count of the contents of each category; lets the user know what size of result set to expect if they choose that facet.
- Generating groupings on the fly, such as size, price or date.
- Drill down by facet, so a record enthusiast could choose genre, artist, title, year.
- Adding special facets within categories -- e.g. a Yellow Pages site would want to show cuisine and location for restaurant listings but not plumbers.
Search run-time
No surprise that the core functionality of the custom search is built on FullTextSqlQuery offered by the MOSS Search API. In contrast with the SearchHiddenObject, I don’t use KeywordQuery class to run property matches.
Processing query string
The factual processing and rendering of the facets follow the same pattern, designed around binding to querystring, response redirection, and postbacks. This approach allows unlimited drill downs through the facets and seamless synchronization with all OOB search web parts.
Url (query string) is parsed and its structure saved into SearchQuery struct. The existing design retains keyword, scope, view, page and collection of properties. A Helper class provides a means of building MOSS SQL select statement out of the structure. The statement is executed against the MOSS search engine and results are eventually saved into the FacetMenu, that builds into the binding source for the UI.
Each click on a facet results in the call to the same helper classes that build a new Url for the redirection and thus a new postback to the server.
Note: next version should provide a caching mechanism for simple drill-down searches.
Bread Crumbs web part uses the same helper classes and the same synchronization between Url and SearchQuery. Because of that the code that generates the redirection Url is quite elegant:
Where I keep the data
Facet structure is the most important data container responsible for the management and rendering of the facets. Facet Menu is a list of Facet Collections where a Facet Collection is a list of Facets. Facet in nutshell is the value of a managed property and the number of its occurrences (hits) within the resultset. Hits are counted precisely but within a given resultset based on the set value of TotalResults. The facet configuration settings from Selected Columns web part property, such as DisplayName and Image are injected into the structure as well to provide a single source for the facet binding.
Facet menu and collection implement INumerable, and that is a mechanism behind sorting. IClonable interface will be used in the next release to provide hits recalculation after exclusions take place.
This is also quite extensible design. Facets can be enriched with new features simply by defining a new property on the data container. In a future release, the Facet Menu also will be made available for web part communication to provide lightweight API for using results of computation elsewhere.
Rendering
Faceted search web parts utilize only CreateChildControls for rendering.
Search facets are built as a DataList (FacetMenu) of Datalists (FacetCollections). Data binding is achieved via use of ITemplate controls that gives the power of late binding through the implementation of public void InstantiateIn(Control container)
Bread crumbs web part also makes use of DataList. The refinement links utilize the same SearchQuery. This provides an elegant way to build a redirection url in one line of code.
- ResetKeyword – query.Keywords = string.Empty;
- ResetFacets – query.Properties.RemoveAll();
- (x) exclude facet - query.Properties.Remove(facetName);
Second thread
When TotalResults in search facets exceeds 500, the web part runs in 2 threads. The synchronous one will process 300, and the async takes care of all remaining facets. This implementation is VERY light weight and does NOT use AJAX. This means that MOSS SP1 is not required to run the faceted search solution, and neither is the AJAX toolkit at prerequisite for deployment. The functionality is based on the ICallbackEventHandler interface, which was implemented in .NET 1.1. The idea for this came from the CKS: Virtual Earth web parts on CodePlex.
The search facets web part is smart enough to spawn the 2nd thread when search returns less than 500 results. Also 2nd thread doesn’t run in the edit mode. Crucial thing to understand is that running a search query returning thousands of results in a single transaction creates a heavy load on a system. I strongly recommend conducting a comprehensive performance/scalability assessment based on (thorough!) usage profiling. That will help to set a TotalResults number as a compromise between performance and accuracy.
Conclusion
The source code has been posted here (Change Set #15904), and it will continue to evolve. I hope that you’ll find Faceted Search to be much more than just a cool feature that extends MOSS 2007's search capabilities. I’ve put quite a few harvested tips and tricks into the web parts for this solution, so take a look at the source code and leverage them as you see fit. And of course, please contribute back to the community by posting any enhancements or extensions in the Faceted Search discussion board or joining the project team (contact me if you're interested).
Separately, the Search Community Toolkit project on CodePlex recently went live. Its primary objective is to be the umbrella project for grouping together the growing number of search oriented open source projects and to drive awareness and involvement from the SharePoint community. For a bit more info about the SCT, read the announcement on the Enterprise Search team blog.
Leonid Lyublinski
Microsoft Consulting Services
