Microsoft Enterprise Search Blog : SharePoint

The Search Developer Story in SharePoint 2010 - Query Interfaces

enterprisesearch — Fri, 20 Nov 2009 23:39:00 GMT

SharePoint 2010 includes a number of features that make the platform easier to use for developers. An improved Visual Studio integration, the addition of LINQ to the SharePoint platform, sandboxing for deployment, and the new developer dashboard are just a few examples of how developing and deploying SharePoint solutions have become much easier.

As a member of the enterprise search development team that has worked to bring FAST Search into SharePoint 2010, I can tell you that a lot has also been done to benefit developers of search-based solutions. SharePoint 2010 Search and the new FAST Search for SharePoint 2010 have been designed to share a common platform so that search developers can integrate with both SharePoint Search and FAST Search for SharePoint 2010 using the same query side interfaces. This means developers don’t have to learn new APIs or programming models, but can leverage the same object models, services and a common query language for both products.

SharePoint developers and architects implementing search-driven applications should understand the available integration options. Depending on requirements, tools, and preferences, one can choose from among several integration points, including a brand new object model in SharePoint 2010. Here’s a list of the different integration points with a brief description of each:

The Federation Object Model (OM)

This is a new search object model in SharePoint 2010. It provides a unified interface for querying against different locations (search providers), giving developers of search-driven Web Parts a way to implement end-user experiences that are independent of the underlying search engine. The object model also allows for combining and merging results from different search providers. Out-of-box Web Parts in SharePoint 2010 are based on this OM, and SharePoint 2010 ships with 3 different types of locations; SharePoint Search, FAST Search and OpenSearch. The Federation OM is also extensible, should you want or need to implement a custom search location outside of the supported types.

The Query Web Service

This is the integration point for applications outside of your SharePoint environment, such as standalone, non-web based applications, or Silverlight applications running in a browser. The Query Web Service is a SOAP based ASMX web service, and supports a number of operations, including:

Querying and getting search results

Getting query suggestions

Getting meta data, e.g. a list managed properties

The same schema is shared for SharePoint Search and FAST Search, and both products support the same operations. For querying, clients can easily switch the search provider by setting a ResultsProvider element in the request XML. A number of extensions are available for FAST Search, e.g. refinement results, advanced sorting using a formula, issuing queries using the FAST Query Language.

The Query RSS Feed

Certain scenarios, like simple mashups, may need only a simple search result list. The RSS feed is an alternative, lightweight integration point for supplying applications outside of SharePoint with a simple RSS result list. The Search Center - the default search front-end in SharePoint 2010 - includes a link to a query-based RSS feed. Switching the engine to the RSS format is done by simply setting a URL provider. Because of its intended simplicity, there are some limitations to what can be returned and customized in the query RSS feed. The object models or the web service integration scenarios are recommended for more advanced applications.

The Query Object Model

This is the lowest level object model, used by the Federation object model, the Query Web Service and the Query RSS feed. Both SharePoint Search and FAST Search support the KeywordQuery object in this object model. While the Federation OM returns XML (to Web Parts), the Query OM returns data types.

The Search Web Parts

Search Web Parts in SharePoint 2010 are common in SharePoint Search and FAST Search, and are now based on the common Federation OM. The Web Parts on a page communicate through a shared Query Manager, a central component of the Federation OM. This makes adding new Web Parts that interact with existing Web Parts simpler than before. For example, a new Tag Cloud Web Part for visualizing the query results can utilize the shared Query Manager for getting results. Developers will also be able to extend out-of-box Web Parts as they now are public in SharePoint 2010 (no longer sealed).

The Common Query Language

Both SharePoint Search and FAST Search support the Keyword Query Language syntax. This is the default query language for both products, and the end-user language supported from the Web Parts in the search centers (including the advanced search page).

FAST Search Extensions

FAST Search has a number of extensions beyond the standard SharePoint Search that are available on both the Federation and Query object models, and as well as on the query web service. Some examples are:

The FAST Query Language, which supports advanced query operators like XRANK for dynamic (query time) term weighting and ranking.
Deep refiners over the whole results set, and the possibility of adding refiners over any managed property
Advanced sorting using managed properties or a query-time sort formula.
Advanced duplicate trimming, with the ability to specify a custom property on which to base duplicate comparisons.
“Similar documents” matching.
FAST Search Admin Object Model for promoting documents or assigning visual best bets to query keywords/phrases.

Building powerful search applications is easier than ever in SharePoint 2010. FAST Search is now integrated into the SharePoint platform and developers of search-driven solutions and applications can leverage a common platform and common APIs for both SharePoint Search and FAST Search. This means applications can be built to support both search engines and then extended if and when desired to take advantage of the more advanced features available with FAST Search, such as dynamic ranking, flexible sort formulae, or deep refiners for insight into your full result set.

Arnt Schøning, Senior Development Engineer | Microsoft Enterprise Search Group

(o on Twitter as @aschoning)

FAST meets SharePoint - What's Coming in Search for SharePoint 2010

ntreloar — Wed, 28 Oct 2009 01:22:00 GMT

Last week was the 2009 SharePoint Conference in Las Vegas. The sold-out attendance of 7400 doubled the number from the previous SharePoint conference 1 ½ years ago. This is not too surprising given the incredible momentum of SharePoint and the fact that much of the event was dedicated to disclosure of the highly anticipated SharePoint 2010 release. Surprising or not, it was gratifying for us search guys to see the level of interest in the new search capabilities being disclosed for 2010. Several of the search-specific break-out sessions had as many people in the audience (>1000) as the entire attendance of our FASTforward’09 search conference in February earlier this year.

At that FASTforward conference in February, Microsoft announced plans to deliver enterprise search targeting two general solution areas: 1) business productivity applications, where the emphasis is on search driving employee efficiency, and 2) Internet business applications, where search is used to drive customer service and revenue. The disclosure of the new search options in SharePoint 2010 at last weeks SharePoint Conference amounts to the first deliverable of this strategy.

SharePoint as a whole has evolved from the original content management and portal platform of earlier releases into a complete “business collaboration platform”, and there are *a lot* of enhancements and new capabilities in SharePoint 2010. I won’t even attempt to summarize them all here. Instead, check out Jeff Teper’s blog post from early last week, which provides a remarkably good summary of everything that’s coming in SharePoint 2010. As Jeff points out in his blog, search is just one of several major categories of capabilities in SharePoint 2010, but “enterprise search is a big investment area for Microsoft” and an area where “we’ve added depth at all levels in 2010”.

There are two main enterprise search options coming with the SharePoint 2010 release:

1) SharePoint Server 2010 Search – the out-of-the-box SharePoint search for enterprise deployments.

2) FAST Search Server 2010 for SharePoint – a brand new add-on product based on the FAST search technology that combines the best of FAST’s high-end search capabilities with the best of SharePoint.

SharePoint Server 2010 Search represents an important upgrade to the existing search for SharePoint, while FAST Search for SharePoint 2010 is a completely new offering and the first new product based on the FAST technology since FAST was acquired by Microsoft in April 2008. Customers and partners familiar with search in previous versions of SharePoint will see many important improvements in 2010, regardless of which product they deploy. For example, there is a new People Search feature for expertise identification and search-driven collaboration, to name just one (see Jeff’s post for a good summary of these general improvements).

FAST Search for SharePoint 2010 adds a whole new level of search capabilities that are a superset of what comes in the out-of-the-box SharePoint 2010 Search option. Since there are now two search options in 2010, it’s useful to understand what is unique in FAST Search for SharePoint and when you might consider using it over the out-of-the-box SharePoint 2010 search. With that in mind, here are my 10 reasons to consider FAST Search for SharePoint 2010:

1) Content Processing Pipeline

For people familiar with the FAST Enterprise Search Platform (ESP), the good news is that the most valued capabilities of ESP have been brought into FAST Search for SharePoint 2010 and made easier to access and deploy through tight integration with the SharePoint management and development tools. The open framework in FAST ESP for creating custom content processing pipelines is a good example. Since it was first introduced in version 3 way back in 2002, FAST customers and partners have leveraged advanced content processing and advanced linguistic features to create a wide variety of novel search applications. This highly valued aspect of the FAST ESP will be available in FAST Search for SharePoint and has been architected and enhanced to take advantage of the SharePoint management interfaces and development tools like PowerShell.

2) Meta-data Extraction

Meta-data is used in search for faceted refinement, relevancy tuning, targeted queries (e.g. search only the authors field), and other general techniques designed to improve findability. The problem is that unstructured documents are often devoid of useful meta-data. The ability to automatically extract meta-data to create useful structure on otherwise unstructured documents is a feature of FAST ESP that will also available in FAST Search for SharePoint 2010. Importantly, FAST Search for SharePoint 2010 takes advantage of simple administrative tools and the concept of “managed properties” in SharePoint to support adding custom meta-data extractors very quickly.

3) Structured Data Search

Structured data search is possible with both search options in SharePoint 2010, but FAST Search for SharePoint 2010 adds an extra level of sophistication for searching data that contains numbers, dates, and other encoded and structured information. To start, the full FAST Query Language (FQL) is available to application developers who want the richness and expressiveness that FQL provides. This includes support for numeric and date data types, formula-based query operations, term weighting with the XRANK operator, and much more. Also, integration with the new Microsoft Business Data Connectivity services in 2010 means that ingesting structured data from external Line of Business applications is much easier in FAST Search for SharePoint.

4) “Deep” Refinement (Faceted Search)

Previously only available in SharePoint search through 3^rd party add-ons, faceted search, called “refiners” in the default search interface (SharePoint Search Center), is now native in the out-of-box SharePoint 2010 search. FAST Search for SharePoint adds to this the ability to deliver faceted search across results sets of any size while retaining precise counts on the refinement facets. This is critical for research and analysis applications where precise counts on facets are important decision making criteria. (You can see examples of deep refiners on FAST ESP powered sites like scirus.com and dell.com.)

5) Visual Search (Document Thumbnails and Previews)

Visual document thumbnails and previewer Web Parts will be out-of-the-box with FAST Search for SharePoint 2010 to help users more quickly judge what is relevant in a search result list. This includes a graphical previewer for PowerPoint presentations based on Microsoft Silverlight that allows users to quickly find the “one slide” of interest without having to open up the entire presentation.

6) Advanced linguistics

The quality of search against text data is highly dependent on the ability to apply the right language-specific processing techniques. FAST Search for SharePoint 2010 builds on the FAST ESP heritage and Microsoft tools to include advanced language processing (linguistics) for dozens of languages, including optimized processing for Chinese/Japanese/Korean.

7) Visual best bets

SharePoint already supports the concept of search Best Bets – managed results delivered with the search for specific queries. FAST Search for SharePoint adds to this the ability to render visual best bests in the form of images and even videos. Management of search best bets, both standard and visual, is through the standard SharePoint administrative console.

8) Best-in-class development platform

FAST Search for SharePoint 2010 builds on the comprehensive development framework of SharePoint 2010. The customization options range from configuring out-of-the-box search behavior (best bets) and user interface controls (Web Parts), to extending existing functionality using public Web Part code and SharePoint Designer, to creating brand new components and functionality with the available APIs. For FAST ESP aficionados, no compromises have been made in the area of extensibility with FAST Search for SharePoint, but many of the customizations in ESP are now much easier to do.

9) Custom search experiences (per user/profile)

FAST Search for SharePoint 2010 includes the same level of relevancy tuning available to FAST ESP. It will be possible, as it is in ESP, to create custom relevancy models tuned to differences in content sources, application needs, and user contexts. User context simply means that different users can have different search “contexts” that enable experiences optimized for their specific business needs. User context can be used to set the search sources, relevance rank profile, linguistic processing features, and other search features by user or user group. In an enterprise search setting, this means that a Sales Director does not have to see the exact same results as a Product Designer for a given query, even if they are searching the same sources.

10) Extreme Scale and Performance

Scale and performance of the out-of-the-box SharePoint 2010 Search has been dramatically improved – with proven scalability to 100 million documents and more. For FAST Search for SharePoint 2010, the exact same scale-out model that exists in FAST ESP has been preserved to enable extremes of content (e.g. number of documents to search), queries (e.g. the number of queries or query rate), or both. This means search solutions that can support billions of documents and thousands of queries per second.

There is much to like about what is coming with search in SharePoint 2010 and more information than I’m able to share in one blog post. You can add to the list above the general benefits of search enhanced by all the other tools and services of the SharePoint platform - including content management, communication and collaboration, business intelligence, system administration and monitoring, application development, and so on. As I’ve pointed out in previous posts, search doesn’t exist in a vacuum, and the ability to integrate and interoperate with other business productivity functions is critical to actually acting on a search result. From this point of view, SharePoint and it’s compendium of integrated services, simply makes search better.

The first public beta for SharePoint 2010 will be available in a few weeks. This will include beta bits for the standard search in SharePoint 2010 and FAST Search for SharePoint 2010. I hope you’ll be able to try out these new search products and features. In the mean time, you can learn more about what’s coming in search for SharePoint 2010 by going to the SharePoint 2010 preview site at http://sharepoint2010.microsoft.com/.

Nate

One Year with Microsoft – a FAST Perspective

ntreloar — Sat, 18 Apr 2009 01:49:00 GMT

After years of writing customer proposals, internal memoranda, and various stuffily formal documents, it feels like a luxury to be able to just write what I think about enterprise search. It’s actually part of my job these days and I’m looking forward to sharing a perspective from 13 years in the industry – the past 6 years with FAST and, most recently, with Microsoft.

As a reminder, it’s been a more than a year since the original offer came down from Microsoft to acquire FAST. To be precise, the bid was announced on January 8^th, 2008 and the deal closed on April 25^th, 2008. The FAST team now makes up a large part of the new Enterprise Search Group (ESG) within the Microsoft Business Division (MBD) – the division that makes SharePoint, the Office line of products, Exchange, etc… .

When I get asked about my reaction to the FAST acquisition by Microsoft, I tend to point out that, while those of us in the business have always understood the value of search, nothing says “Ata boy!” like having the largest software company in the world take notice. Maybe we could ask why it took so long, but even if you didn’t happen to work at FAST, you can’t help but feel that Microsoft’s move is validation of our growing corner of the IT industry.

I admit that the answer above, while maybe heartwarming, doesn’t get to the core of what people really want to know. Not surprisingly, folks are more interested in Microsoft’s vision for enterprise search and plans for the FAST people, products, partners, and customers than they are in my emotions. Now, with a year under the belt at Microsoft, I have a few more insights to offer than just the initial “nice validation” response.

In his keynote presentation at FASTforward’09 in February, Kirk Koenigsbauer addressed three key topics related to Microsoft’s interest in enterprise search (a transcript of Kirk’s keynote can be found here). These were:

· Commitment (to enterprise search)

· Vision

· Product Plans

These topics provide a useful framework for sharing my own observations.

Commitment

There are a number of anecdotal facts that point to Microsoft’s commitment to being a leader in enterprise search. Kirk shared a few of these in his keynote – things like the percentage of Microsoft Research investment going to search (appx 15%), the size of the Enterprise Search Group R&D organization (several hundred engineers and growing), and of course the investment itself to acquire FAST (US$1.2B). There are other supporting data points, like the announcement of Oslo (FAST’s headquarters) as a key R&D center for business search.

Any one of these facts is a strong indication of Microsoft’s ambitions in this space, but my take is that the evidence of Microsoft’s commitment to search comes from more than these metrics or executive statements. It comes from a growing grass roots interest in search across all of Microsoft. For example, I often get a question like this from customers and partners:

“Have you guys talked with the folks over in Microsoft’s <product name> team?”

…and then…

“ Man, you should because FAST technology added to what they’re doing would be powerful combination.”

The usual answer is, yes, we’ve talked to the <product name> team and, yes, there are some very interesting ideas and even some specific activity that we mostly can’t talk about yet. In fact, what’s been most interesting and fun for us former FAST folks is the breadth of technologies that we can now include in our conversations with customers and partners. SharePoint is the “hero SKU”, as we say here, and the combination of FAST search with the capabilities of SharePoint makes for an impressive offering for both intranet and Internet applications that are focused on helping people consume and use information. It’s not a leap to recognize that Microsoft has something to offer at almost every level of an IT solution “stack” complementing the capabilities of both SharePoint and search – from the operating system to application development tools and even cloud-based services. To put it in perspective, ask yourself how many companies offer both a world class enterprise search platform and a world class relational database.

To be honest, search is such a generally valued concept and the possibilities are so compelling when it’s combined with other Microsoft products and technology that it’s all we can do to stay focused on our main priorities. It’s a good problem.

Vision

At some point prior to the acquisition, the Microsoft enterprise search team came to a vision of search that matched what we had developed at FAST. Specifically, that search is more than just a search box and a list of blue document links, but represents a set of capabilities that are enabling new ways to engage users by creating personalized, conversational experiences that cater to the way people prefer to consume and interact with information. This vision was behind the principle theme for the FASTforward’09 conference this past February – “Engage Your Users”.

Whether the original Microsoft team came to this vision independently or after talking to FAST folks (ego would like to think the latter) is less important than the fact that it is now a shared vision throughout the Microsoft Enterprise Search Group and is shaping how we are investing in product development. It’s also a vision that is permeating into other areas within Microsoft. For example, I recently had a chance to apply this way of thinking about search to some other very interesting Microsoft technology, Microsoft Surface, but that’s a topic for another post.

Product Plans

At FASTforward’09 we announced our plans to target enterprise search in two areas:

· Business productivity – applications inside the firewall where, in particular, SharePoint provides the framework for content management and collaboration.

· Internet business – “outside the firewall” applications for attracting, retaining, and otherwise monetizing customers.

The intentions are to have a common search platform supporting both of these general markets and to include application specific capabilities and templates that are unique to each. FAST had already started down this path. For example, FAST AdMomentum is an ad platform that interoperates with search and is relevant to monetization strategies in Internet Businesses, but not so obvious of a fit for inside the firewall apps.

This relatively straightforward strategy and message was very important to get out to the FAST customers base, especially given that Internet Businesses have made up well more than half of FAST’s business to date. Also, most industry pundits will tell you that the requirements for search inside the corporate firewall are simply different than search in consumer facing applications. Even so, what’s so promising to me about this strategy is that there are opportunities to “bleed” capabilities between these two application spaces. We saw this “consumerization” of search features happen more than once at FAST. Features that we initially designed for consumer search found their way into intranet search deployments (one simple example is the “best bets” concept like the one found in SharePoint). The opposite has also happened. Now, consider the capabilities in SharePoint, which is already powering many consumer facing Web sites, and you can see where this can lead.

There you have it, my first post for the Microsoft Enterprise Search Blog. Look for more posts from me in this general category of enterprise search vision and strategy. I welcome all comments on this and future entries.

Next up – Search plus Natural User Interfaces.

Nate

Microsoft positioned in the Leaders Quadrant of the 2008 Information Access Magic Quadrant

enterprisesearch — Fri, 31 Oct 2008 00:33:00 GMT

We’ve got great news to share! Last month, Gartner published the 2008 Magic Quadrant for Information Access Technology, and Microsoft was positioned in the Leaders Quadrant. Since the completion of the acquisition, we’ve worked incredibly hard to communicate and demonstrate a combined vision and strategy to our customers and partners. It’s good to know we’re heading in the right direction!

When I talk with customers about search, it’s clear that organizations have very different needs. In fact, many people tell me that even within an organization the one-size-fits-all approach just doesn’t work. So over the last year, we’ve announced some bold moves designed to create a compelling portfolio of search applications. With the addition of Search Server Express and the acquisition of FAST, we now have a product line-up designed to meet a broad range of business needs:

Some departments or small organizations need search that is quick and easy to set up; we offer Microsoft Search Server Express as a free download so that you can get it up and running in about 30 minutes. We’re excited to see customers like St. Jude Medical and Urbis having quick successes with Express. We’re also seeing partners, such as StartReady, build solutions around Search Server Express to create a search appliance.
Many organizations need search as an integral part of a business productivity infrastructure; Search in Microsoft Office SharePoint Server is integrated with other key SharePoint productivity workloads such as portals, collaboration, ECM, business processes and BI. Customers like McCann Worldgroup and Jones Lang LaSalle are all deriving productivity increases with better search in SharePoint. In particular, both companies are promoting collaboration and leveraging in-house experts with people search enhanced by user profiles in MySites.
Some organizations face business problems that demand high-end search; FAST ESP offers best-in-class search with extreme scalability, query performance, and other advanced capabilities for sophisticated customer-facing or inside-the-firewall applications. For example, Aerotek and TEKsystems, two of the world’s largest staffing companies, deliver job searching to more than 1.3 million users. In more than 164 million queries, greater than 99.5% of query results came back in less than 2 seconds. For inside-the-firewall productivity, they index more than 10 million complex candidate records with low latency during high volume index updates. We’re also excited to see Pfizer pushing the envelope with an Enterprise Collaboration Framework driven by FAST ESP on top of SharePoint

While our “Leaders Quadrant” position in the Magic Quadrant is an important milestone, we still think of this as the very beginning of our journey. We’re continuing to combine our deep technical expertise with our broad reach to deliver exciting innovations to the market – so you can and should expect great things to come. Stay tuned!

Kirk Koenigsbauer
General Manager,
SharePoint Business Group

Magic Quadrant for Information Access Technology (Gartner Research, Sept. 30, 2008) Microsoft is positioned in the Leaders Quadrant of Gartner, Inc.'s 2008 Magic Quadrant for Information Access Technology. This report assesses vendors with capabilities that go beyond enterprise search to encompass a range of technologies. Their capabilities include search; federated search, content classification, categorization and clustering; fact and entity extraction; taxonomy creation and management; information presentation (for example, visualization) to support analysis and understanding; and desktop search to address user-controlled repositories in order to locate and "invoke" documents, data, e-mail and intelligence.

The Magic Quadrant is copyrighted 2008 by Gartner, Inc. and is reused with permission. The Magic Quadrant is a graphical representation of a marketplace at and for a specific time period. It depicts Gartner's analysis of how certain vendors measure against criteria for that marketplace, as defined by Gartner. Gartner does not endorse any vendor, product or service depicted in the Magic Quadrant, and does not advise technology users to select only those vendors placed in the "Leaders" quadrant. The Magic Quadrant is intended solely as a research tool, and is not meant to be a specific guide to action. Gartner disclaims all warranties, express or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

Taking People Search on the Road….

enterprisesearch — Thu, 16 Oct 2008 20:33:53 GMT

In another great blog post Matt McDermott walks you through the steps of enabling SharePoint’s people search capability on a mobile device with the end results looking something like this;

The post is here;

http://blogs.catapultsystems.com/matthew/archive/2008/09/27/mobile-people-search.aspx

Richard Riley
Senior Technical Product Manager
Microsoft Corp.

SharePoint Image Search

enterprisesearch — Sat, 20 Sep 2008 01:03:43 GMT

Matthew McDermott, a SharePoint MVP, has written a great 4 part blog post on how to make SharePoint 2007 search (and Search Server) render image results in a way that looks very similar to http://images.live.com.

Not only does this make searching images much easier, it’s also a very thorough step-by-step tutorial on how to customize results using the built in Web Parts and XSL – it’s well worth a read.

SharePoint Image Search (Part 1)

SharePoint Image Search (Part 2)

SharePoint Image Search (Part 3)

SharePoint Image Search (Part 4)

The end result makes SharePoint Image results look like the screencap below.

Richard Riley
Senior Technical Product Manager
Microsoft Corp.

Announcing Faceted Search v2.5

enterprisesearch — Wed, 13 Aug 2008 00:50:22 GMT

Starting Faceted Search 2.5, the solution relies on Microsoft Enterprise Library to address common software requirements in caching, logging, exception handling, policy injection etc., etc. More importantly, the 2.5 is a ground breaking release that is setting new targets for the Faceted Search. So, what’s new?

New Features

1. Caching – dramatically improves performance and decreases the load on the search engine

The solution uses 2 mechanisms for manageable cache: quick and long. I built the caching logic on assumption that user knows what he/she is looking for. The Search Facets web part will cache original result set and use it for the search refinement, paging and other postbacks. If the initial result set doesn’t provide full coverage of the search, the smart 2^nd thread will run against real-time data providing adjustment to the cached match.

2. Synchronization with Core Search Results web part

The MOSS search is adjusted by several parameters that designer can set for the Core Search Results web part itself. These include remove duplicates, enable trimming, permit noise words. When you drop the Search Facets web part to the search results page, it will find the Core Search Results, read its parameters and sync the search query parameters to exactly match ones used by the Core.

3. Support for advanced search

It was the most wanted feature since Faceted Search 1.0. With 2.5, the Facets are rendered for advanced search although do not extend yet to ranges. The functionality is accomplished by extending SearchQuery structure to accommodate POST requests and sync back to GET query.

4. Match of search counters

This release introduced an updated search syntax that is design to provide matching counters to the core search. In fact, the new search query is using both KeywordQuery and FullTextQuery through the use of generics.

public class GenericQuery<T> : IDisposable where T : Query
{
    private EventHandler _customLogic;

    public ResultTableCollection Execute(EventArgs args)
    {
        _customLogic(_query, args);
        return _query.Execute();
    }

    ...
}

Additionally, the WHERE clause of the search query was modified to provide closer match to the Core counter.

5. Introducing Parent-Child relationships

By design, the facets can support only 2 levels. This release extended the Facets schema to allow management of the nested layers. That eases the pain of displaying complex hierarchies such as geography, or org chart etc. Parent-Child relationship can be set by facet name and facet value, or just by facet name.

<Column Name="BDCCity" DisplayName="City" ParentName="BDCState" />
<Column Name="BDCState" DisplayName="State" >
  <Mappings>
    <Mapping Match="Alberta"  ParentName="BDCCountry" ParentValue="Canada"/>    
    <Mapping Match="Manitoba" ParentName="BDCCountry" ParentValue="Canada" />
    <Mapping Match="Ontario"  ParentName="BDCCountry" ParentValue="Canada"/>
    <Mapping Match="Quebec"   ParentName="BDCCountry" ParentValue="Canada"/>
  </Mappings>
</Column>

In the configuration above, the City facets will display only after the user chose the State. The State itself will match the country of origin.

6. Extending search to logical “OR” queries

Original facets always represent “AND” queries. That implies ability to narrow the search results by adding extra criteria. In this release I prototyped the way to expand the search by adding additional matches to the criteris. This in fact resulted in rewamped the Bread Crumbs UI. Proviuded now out-of-the-box support for languages is a good example of how “OR” queries empower the search.

7. Simplified web part properties

The 2.5 release is friendly to modifications of the web part properties. I have all properties classified and broken down to groups for each of the web parts.

8. Other

There are lots and lots of numerous fixes and enhancements, including improved security validation, code refactoring, extending facet sorting, support of quoted search and duplicates etc., etc.

What’s next

It’s my privilege to say that we have a team now that helps to shape new releases and brainstorm the furutre of the Faceted Search. In present we are looking at AJAX and SilverLight and hopefully you’ll start seeing more and more power of Facets in the near future.

Leonid Lyublinski
Senior Consultant
Microsoft Consultancy Services

Announcing: Availability of Infrastructure Updates

enterprisesearch — Tue, 15 Jul 2008 10:08:13 GMT

As announced on the SharePoint Team blog this morning we released to web three new important updates that affect SharePoint Server 2007, Windows SharePoint Services 3.0, Project Server 2007, Search Server 2008, Search Server 2008 Express and Project Professional 2007.

The Infrastructure Update for Microsoft Office Servers (KB951297) (Download X86, Download X64) is particularly important from a Search and SharePoint Server 2007 perspective as it contains the new Enterprise Search features that were shipped in Search Server 2008 and Search Server 2008 Express that were are not already in SharePoint Server 2007; this includes Federated Search capability, a unified administration dashboard and several Search core platform performance updates.

For an overview of the new federation features please check out this short video which covers how to configure a federated location and configure one of the new federated search Web Parts to show results from that location.

There’s also a growing number of articles on TechNet and MSDN that cover configuring and troubleshooting federation and extending federation with Federated Search Connectors.

The screen capture below shows how federated search results show up on a results page – the results on the right hand side and top left are federated results, the ones at the bottom left are from the local index.

The new Search Administration Dashboard consolidates all of the Search related admin activities into a single place – there’s also some new functionality in the dashboard (There’s greater granularity for content source crawl history and a convenient list that shows currently running crawls and durations for example) and it makes the Search Administrators job much easier by keeping everything close at hand.

The UI looks like the screen capture below:

The update leaves the old Search admin pages intact, the links to them stay in Central Admin (Along with a new link to the new dashboard) so if you’ve made any changes to them or just prefer to use the existing admin pages you’re free to do so.

The other changes are all under the hood and improve Index and Query performance as well as fixing a few bugs. Check out the KB articles below for more details.

Description of the Infrastructure Update for Microsoft Office Servers (KB951297)

Fixes Included in the Infrastructure Update for Microsoft Office Servers (KB953750)

Please read this post on the SharePoint Team blog and the installation instructions thoroughly before you install the Infrastructure Update for Microsoft Office Servers (KB951297) and the Infrastructure Update for Windows SharePoint Services 3.0 (KB951695) on SharePoint Server 2007 or Search Server 2008.

Install the Infrastructure Update for Microsoft Office Servers (Office SharePoint Server 2007)

Install the Infrastructure Update for Microsoft Office Servers (Search Server 2008)

Finally, if you’re wondering why the Infrastructure Update for Microsoft Office Servers (KB951297) and the Infrastructure Update for Windows SharePoint Services 3.0 (KB951695) applies to Search Server 2008 then you’re probably not alone!

There’s a very good reason for both – Search Server 2008 and Search Server 2008 Express are built on the Windows SharePoint Services 3.0 platform (hence the need for the Infrastructure Update for Windows SharePoint Services 3.0 (KB951695)) and secondly the Search features are from SharePoint Server 2007 (hence the need for the Infrastructure Update for Microsoft Office Servers (KB951297)). The latter update includes a few bug fixes since Search Server 2008 and Search Server 2008 Express launched.

We strongly recommend that you install the updates that apply to you as soon as your patching and maintenance schedules permit.

Richard Riley
Senior Technical Product Manager
Microsoft Corp.

Announcing: conceptClassifier for SharePoint

enterprisesearch — Tue, 08 Jul 2008 00:41:07 GMT

conceptClassifier for SharePoint adds automatic document classification and taxonomy management to Microsoft SharePoint and works without the need to build another search index. It is installed as a set of Features that, when activated, cause new columns to be displayed in the document library listings and new menu options appear that allow authorised users to edit the automatically generated metadata, if required.

Adding Taxonomy navigation to SharePoint

Classification results are saved directly into SharePoint Properties where Microsoft Enterprise Search can utilise the metadata for enhanced searching, such as faceted search or results filtering.

The accuracy of the automatic classification is driven by the underlying technology which is based on compound term processing. This means that the classification engine performs its matching using multi-word concepts rather than simply looking for selected keywords or phrases. Taxonomy creation and maintenance is a simple process and is conducted using natural language rules, making it much simpler and quicker than alternative approaches.

More information about conceptClassifier for SharePoint can be found here:

http://www.conceptSearching.com

and a SharePoint demonstration can be seen here:

http://moss.conceptSearching.com

John Challis
CTO
Concept Searching Limited

Announcing: SharePoint Web Parts for FAST ESP

enterprisesearch — Fri, 20 Jun 2008 20:06:00 GMT

It’s been around 45 days since the acquisition of FAST Search and Transfer closed and we’re moving quickly to provide interoperability for Microsoft customers between FAST ESP and Microsoft SharePoint Server.

The first deliverables from this work are a set of FAST ESP Search Web Parts for quickly integrating results from FAST ESP into SharePoint Server 2007 and a FAST ESP Search site template.

Using these Web Parts and Site Template SharePoint administrators will be able to quickly and easily build FAST ESP-based search sites inside SharePoint 2007 by simply dropping in and configuring the appropriate components.

The Web Parts and Site Template are available as a free download (both compiled code and source code) from CodePlex at www.codeplex.com/espwebparts and are part of the Search Community Toolkit.

Some of the FAST ESP search capabilities that can be exposed within SharePoint Server 2007 using these Web Parts include:

• Search Box Web Part -- Search box for query term submission and includes “did you mean” functionality for query correction

• Result List Web Part -- Displays search results and supports sorting, pagination, and navigator-based filtering

• Navigator Web Part -- Displays dynamic navigators that profile search results across a set of pre-defined dimensions and allow users to refine the search through navigation clicks

• Breadcrumb Web Part -- Displays the search term(s) and list of navigators used to obtain the current result set

The FAST ESP Web parts are designed to be open and extensible, and we’re actively encouraging customers and partners to download them, customize them to align with their branding and extend them to fit their search and user experience requirements.

Expect the features, functionality and range of ESP Web Parts to grow through contributions from the search developer community as well as further contributions from the FAST & Microsoft Search Team!

FAST & Microsoft Search Teams.

Indexing Exchange Server 2007 Public Folders

enterprisesearch — Sat, 07 Jun 2008 03:56:42 GMT

I've had several questions recently about how to index Exchange Server 2007 Public Folders with SharePoint Server 2007.

Unfortunately with the RTM versions of both products it's not actually possible due a couple of issues with both Exchange Server 2007 and SharePoint Server 2007.

The good news however, is that everything is back in working order if you install Exchange Server 2007 Service Pack 1 *and* your SharePoint Server also has Service Pack 1 installed.

The RTM versions of Search Server 2008 and Search Server 2008 Express are unaffected (As they already include the Service Pack 1 changes), so providing Exchange Server 2007 has Service Pack 1 installed they will both work.

None of this affects Exchange Server 2003 which works with the RTM versions SharePoint Server 2007, Search Server 2008 and Search Server 2008 Express and SharePoint Server 2007 with SP1.

Hopefully this stops people scratching their heads...

Richard Riley
Senior Technical Product Manager
Microsoft Corp.

Introducing Protocol Handler.NET

enterprisesearch — Thu, 05 Jun 2008 09:13:27 GMT

"Protocol Handler.Net is a set of .Net wrappers for the protocol handler interfaces that enable developers to create and deploy protocol handlers for SharePoint search and Search Server.

Developers can index data and documents from any system they can connect to.

Much of the complexity and time around the development of protocol handlers, such as COM interoperability, are reduced and hidden in the wrappers themselves letting developers just concentrate on code to connect to a content source and pull data.

Protocol Handler.Net makes it possible to develop protocol handlers in C# or VB.Net and simplifies the handling of security, metadata, streaming content, deployment and management just to name a few things. It also comes with a help system and sample project to further help developers."

Big thank you to to Chris Gomez from http://www.FastSharePoint.com for creating and sharing these tools and samples!

They are available now on CodePlex at http://www.codeplex.com/phdotnet and are part of the Search Community Toolkit

Announcing: Release to Web of Documentum and FileNet Indexing Connectors

enterprisesearch — Wed, 28 May 2008 03:45:46 GMT

Today marks the release of 2 new Microsoft Enterprise Search Indexing Connectors (formally known as Protocol Handlers) for EMC Documentum 5.3 (Service Pack 4) and IBM FileNet P8 3.5.1 or 3.5.2.

The connectors are compatible with the 32bit English Language versions of SharePoint Server 2007 (Service Pack 1), Search Server 2008 and Search Server 2008 Express, and are available as FREE downloads from;

Enterprise Search Indexing Connector 2008 for EMC Documentum

Enterprise Search Indexing Connector 2008 for IBM FileNet

Installation and configuration documentation is included in the download and the release notes are available here;

Release Notes for Indexing Connector 2008 for EMC Documentum

Release Notes for Indexing Connector 2008 for IBM FileNet

A couple of overview videos to get you up and running with the connectors quickly are available through the links below;

Overview Video - Installing and configuring the EMC Documentum Connector

Overview Video - Installing and configuring the IBM FileNet Connector

Both of these connectors are fully supported through your existing service contract with Product Support Services or through the regular pay per incident channel.

Introducing the ExportCrawlLog STSADM Command Extension

enterprisesearch — Tue, 27 May 2008 06:46:21 GMT

In versions of SharePoint prior to MOSS 2007 each time a crawl was executed, a new group of log messages were stored to the database. Also, the name of the log itself was changed in the documentation and the user interface. Formerly this log was known as the Gatherer Log, but it is now called the Crawl Log.

When troubleshooting problems with the crawl of a particular content source it was (and still is) sometimes useful to compare and contrast the messages logged between one crawl and the next. In MOSS 2007, the storage of the crawl log messages has been optimized/minimized such that only the most recent message for a given URL is stored in the database. As a consequence, the results from a prior crawl are overwritten by results from subsequent crawls. In other words, you can only ever see the most recent log message for a given URL.

This is where the STSADM command extension “ExportCrawlLog” comes in. The motivation for preparing this tool is to provide a way to make a “snapshot” of the Crawl Log information at a point in time to facilitate post-mortem analysis of crawl problems. As a bonus, in addition to extracting crawl log detail, it also provides some summary reporting features. The goal of the tool is to provide a means of gathering data by which you can track and manage the health of your index over time. For instance you could setup a scheduled task to run this command once a day and generate summary reports that can provide data for trend monitoring.

ExportCrawlLog uses only the published APIs of the SharePoint Object Model and must be run on the index server of your SharePoint Farm. ExportCrawlLog is available as source code on Codeplex at http://www.codeplex.com/ExportCrawlLog and is part of the Search Community Toolkit.

Please use the Discussion tracking and Issue tracking features of Codeplex to offer your feedback.

Larry Kuhn
Architect
Microsoft Consulting Services.

Understanding Total Hits & Paging in the MOSS 2007 Search API

enterprisesearch — Thu, 22 May 2008 21:45:06 GMT

One of the more discussed topics I’ve seen (and struggled with myself) is around the concept of obtaining the total number of hits in a search results when working with the MOSS API. For instance, when I search for “sales forecast” in my SharePoint site, I want to not only see a set of paged results, 10 hits per page, but also see that my search found 127 matches. Those of you who’ve worked with the SharePoint Search Web Parts know this is a piece of cake using the Search Core Results, Search Paging and Search Statistics Web Parts.

But what if you need to roll your own solution? How can you get the same data out of your search query using the MOSS 2007 Search API? At first this can be a bit tricky but this post will hopefully show you how to knock it down to being a trivial task.

Executing a search query against the SharePoint API has you working with two objects that implement the abstract class Microsoft.Office.Server.Search.Query.Query: Microsoft.Office.Server.Search.Query.KeywordQuery and Microsoft.Office.Server.Search.Query.FullTextSqlQuery. The former KeywordQuery is useful for simple queries whereas the latter FullTextSqlQuery is much more powerful. Both implement the Execute() method which executes the defined query and returns back a collection of results as type Microsoft.Office.Server.Search.Query.ResultTableCollection. Using this object, you can get the specific results you are interested in. For instance to get the relevant results use the following to get an instance of a specific Microsoft.Office.Server.Search.Query.ResultTable:

using (FullTextSqlQuery query = new FullTextSqlQuery(SPContext.Current.Site))

{

query.QueryText = "SELECT Rank, Title Url FROM Scope() WHERE FREETEXT(defaultproperties,'sales proposal') ORDER BY Rank Desc",

ResultTableCollection results = query.Execute();

ResultTable relevantResults = results[ResultType.RelevantResults];

// do work with the results

}

Simple enough, but the project requires much more than that as usual. What we need to do is page the results to show only 15 items per page. No problem… let’s just modify that query a bit to set the Query.StartRow & Query.RowLimit properties of the query to say what page we’re on and tell SharePoint how many results we want to get back. Take for instance if we’re on page 2 of the results… we want to start with the 16^th hit as 1-15 were on page 1:

using (FullTextSqlQuery query = new FullTextSqlQuery(SPContext.Current.Site))

{

query.StartRow = 16;

query.RowLimit = 15;

query.QueryText = "SELECT Rank, Title Url FROM Scope() WHERE FREETEXT(defaultproperties,'sales proposal') ORDER BY Rank Desc",

ResultTableCollection results = query.Execute();

ResultTable relevantResults = results[ResultType.RelevantResults];

// do work with the results

}

Again… pretty straight forward. Now is where it gets a bit tricky. You need to show links to provide paging… but in order to do that you need a good idea what the total reset set of your search query because if there were only 43 hits, you don’t want to shot options to jump to page 9. The property that gives you the number you’re looking for is ResultTable.TotalResults. Now there’s something special about this guy: he doesn’t give you an exact number… he gives you an estimate. Why an estimate? Quite simply, with all the security trimming and other complex logic inherit to search algorithms, it’s just too expensive to get a specific number. Sites like Live.com can do this because they don’t have to concern themselves with the security trimming of hits.

But this is not all… there’s another property you should pay attention to: Query.TotalRowsExactMinimum. This property tells SharePoint this is the minimum number of hits to be included in the search. It’s used to generate the estimate of total results. Think of it like a hint to search… saying “you only have to work this hard on this query.” Most search implementations only show the next few paging options… they don’t show ALL the options. For instance, if you’re on page 5, your paging control may show the following:

«Previous« 2 3 4 5 6 7 8 »Next»

In this case, you don’t need for search to find ALL the results… you only need it to determine how many more page options you want to show to see if you’re going to show too many or too few. In the above example, you have an additional 3 pages of results you want to show. Continuing on this example, you have a result set of 15 and you have an additional 3 pages you want to show, the Query.TotalResultsExactMinimum property would be 45 as it already is going to factor into the equation the Query.StartRow property:

using (FullTextSqlQuery query = new FullTextSqlQuery(SPContext.Current.Site))

{

query.StartRow = 16;

query.RowLimit = 15;

// TotalRowsExactMinimum = [number of pages to show] * [page size]

query.TotalRowsExactMinimum = 45;

query.QueryText = "SELECT Rank, Title Url FROM Scope() WHERE FREETEXT(defaultproperties,'sales proposal') ORDER BY Rank Desc",

ResultTableCollection results = query.Execute();

ResultTable relevantResults = results[ResultType.RelevantResults];

// do work with the results

}

That’s all there really is to it! One parting word of advice: use the Query.TotalRowsExactMinimum property with care as the higher its set, the greater performance impact there will be on each search query executed.

A special shout out & thanks to Puneet Narula @ Microsoft for helping uncover this very helpful nugget of info.

Andrew Connell (blog)
Microsoft MVP