Microsoft Enterprise Search Blog : Updates

FAST meets SharePoint - What's Coming in Search for SharePoint 2010

ntreloar — Wed, 28 Oct 2009 01:22:00 GMT

Last week was the 2009 SharePoint Conference in Las Vegas. The sold-out attendance of 7400 doubled the number from the previous SharePoint conference 1 ½ years ago. This is not too surprising given the incredible momentum of SharePoint and the fact that much of the event was dedicated to disclosure of the highly anticipated SharePoint 2010 release. Surprising or not, it was gratifying for us search guys to see the level of interest in the new search capabilities being disclosed for 2010. Several of the search-specific break-out sessions had as many people in the audience (>1000) as the entire attendance of our FASTforward’09 search conference in February earlier this year.

At that FASTforward conference in February, Microsoft announced plans to deliver enterprise search targeting two general solution areas: 1) business productivity applications, where the emphasis is on search driving employee efficiency, and 2) Internet business applications, where search is used to drive customer service and revenue. The disclosure of the new search options in SharePoint 2010 at last weeks SharePoint Conference amounts to the first deliverable of this strategy.

SharePoint as a whole has evolved from the original content management and portal platform of earlier releases into a complete “business collaboration platform”, and there are *a lot* of enhancements and new capabilities in SharePoint 2010. I won’t even attempt to summarize them all here. Instead, check out Jeff Teper’s blog post from early last week, which provides a remarkably good summary of everything that’s coming in SharePoint 2010. As Jeff points out in his blog, search is just one of several major categories of capabilities in SharePoint 2010, but “enterprise search is a big investment area for Microsoft” and an area where “we’ve added depth at all levels in 2010”.

There are two main enterprise search options coming with the SharePoint 2010 release:

1) SharePoint Server 2010 Search – the out-of-the-box SharePoint search for enterprise deployments.

2) FAST Search Server 2010 for SharePoint – a brand new add-on product based on the FAST search technology that combines the best of FAST’s high-end search capabilities with the best of SharePoint.

SharePoint Server 2010 Search represents an important upgrade to the existing search for SharePoint, while FAST Search for SharePoint 2010 is a completely new offering and the first new product based on the FAST technology since FAST was acquired by Microsoft in April 2008. Customers and partners familiar with search in previous versions of SharePoint will see many important improvements in 2010, regardless of which product they deploy. For example, there is a new People Search feature for expertise identification and search-driven collaboration, to name just one (see Jeff’s post for a good summary of these general improvements).

FAST Search for SharePoint 2010 adds a whole new level of search capabilities that are a superset of what comes in the out-of-the-box SharePoint 2010 Search option. Since there are now two search options in 2010, it’s useful to understand what is unique in FAST Search for SharePoint and when you might consider using it over the out-of-the-box SharePoint 2010 search. With that in mind, here are my 10 reasons to consider FAST Search for SharePoint 2010:

1) Content Processing Pipeline

For people familiar with the FAST Enterprise Search Platform (ESP), the good news is that the most valued capabilities of ESP have been brought into FAST Search for SharePoint 2010 and made easier to access and deploy through tight integration with the SharePoint management and development tools. The open framework in FAST ESP for creating custom content processing pipelines is a good example. Since it was first introduced in version 3 way back in 2002, FAST customers and partners have leveraged advanced content processing and advanced linguistic features to create a wide variety of novel search applications. This highly valued aspect of the FAST ESP will be available in FAST Search for SharePoint and has been architected and enhanced to take advantage of the SharePoint management interfaces and development tools like PowerShell.

2) Meta-data Extraction

Meta-data is used in search for faceted refinement, relevancy tuning, targeted queries (e.g. search only the authors field), and other general techniques designed to improve findability. The problem is that unstructured documents are often devoid of useful meta-data. The ability to automatically extract meta-data to create useful structure on otherwise unstructured documents is a feature of FAST ESP that will also available in FAST Search for SharePoint 2010. Importantly, FAST Search for SharePoint 2010 takes advantage of simple administrative tools and the concept of “managed properties” in SharePoint to support adding custom meta-data extractors very quickly.

3) Structured Data Search

Structured data search is possible with both search options in SharePoint 2010, but FAST Search for SharePoint 2010 adds an extra level of sophistication for searching data that contains numbers, dates, and other encoded and structured information. To start, the full FAST Query Language (FQL) is available to application developers who want the richness and expressiveness that FQL provides. This includes support for numeric and date data types, formula-based query operations, term weighting with the XRANK operator, and much more. Also, integration with the new Microsoft Business Data Connectivity services in 2010 means that ingesting structured data from external Line of Business applications is much easier in FAST Search for SharePoint.

4) “Deep” Refinement (Faceted Search)

Previously only available in SharePoint search through 3^rd party add-ons, faceted search, called “refiners” in the default search interface (SharePoint Search Center), is now native in the out-of-box SharePoint 2010 search. FAST Search for SharePoint adds to this the ability to deliver faceted search across results sets of any size while retaining precise counts on the refinement facets. This is critical for research and analysis applications where precise counts on facets are important decision making criteria. (You can see examples of deep refiners on FAST ESP powered sites like scirus.com and dell.com.)

5) Visual Search (Document Thumbnails and Previews)

Visual document thumbnails and previewer Web Parts will be out-of-the-box with FAST Search for SharePoint 2010 to help users more quickly judge what is relevant in a search result list. This includes a graphical previewer for PowerPoint presentations based on Microsoft Silverlight that allows users to quickly find the “one slide” of interest without having to open up the entire presentation.

6) Advanced linguistics

The quality of search against text data is highly dependent on the ability to apply the right language-specific processing techniques. FAST Search for SharePoint 2010 builds on the FAST ESP heritage and Microsoft tools to include advanced language processing (linguistics) for dozens of languages, including optimized processing for Chinese/Japanese/Korean.

7) Visual best bets

SharePoint already supports the concept of search Best Bets – managed results delivered with the search for specific queries. FAST Search for SharePoint adds to this the ability to render visual best bests in the form of images and even videos. Management of search best bets, both standard and visual, is through the standard SharePoint administrative console.

8) Best-in-class development platform

FAST Search for SharePoint 2010 builds on the comprehensive development framework of SharePoint 2010. The customization options range from configuring out-of-the-box search behavior (best bets) and user interface controls (Web Parts), to extending existing functionality using public Web Part code and SharePoint Designer, to creating brand new components and functionality with the available APIs. For FAST ESP aficionados, no compromises have been made in the area of extensibility with FAST Search for SharePoint, but many of the customizations in ESP are now much easier to do.

9) Custom search experiences (per user/profile)

FAST Search for SharePoint 2010 includes the same level of relevancy tuning available to FAST ESP. It will be possible, as it is in ESP, to create custom relevancy models tuned to differences in content sources, application needs, and user contexts. User context simply means that different users can have different search “contexts” that enable experiences optimized for their specific business needs. User context can be used to set the search sources, relevance rank profile, linguistic processing features, and other search features by user or user group. In an enterprise search setting, this means that a Sales Director does not have to see the exact same results as a Product Designer for a given query, even if they are searching the same sources.

10) Extreme Scale and Performance

Scale and performance of the out-of-the-box SharePoint 2010 Search has been dramatically improved – with proven scalability to 100 million documents and more. For FAST Search for SharePoint 2010, the exact same scale-out model that exists in FAST ESP has been preserved to enable extremes of content (e.g. number of documents to search), queries (e.g. the number of queries or query rate), or both. This means search solutions that can support billions of documents and thousands of queries per second.

There is much to like about what is coming with search in SharePoint 2010 and more information than I’m able to share in one blog post. You can add to the list above the general benefits of search enhanced by all the other tools and services of the SharePoint platform - including content management, communication and collaboration, business intelligence, system administration and monitoring, application development, and so on. As I’ve pointed out in previous posts, search doesn’t exist in a vacuum, and the ability to integrate and interoperate with other business productivity functions is critical to actually acting on a search result. From this point of view, SharePoint and it’s compendium of integrated services, simply makes search better.

The first public beta for SharePoint 2010 will be available in a few weeks. This will include beta bits for the standard search in SharePoint 2010 and FAST Search for SharePoint 2010. I hope you’ll be able to try out these new search products and features. In the mean time, you can learn more about what’s coming in search for SharePoint 2010 by going to the SharePoint 2010 preview site at http://sharepoint2010.microsoft.com/.

Nate

Announcing: Availability of Infrastructure Updates

enterprisesearch — Tue, 15 Jul 2008 10:08:13 GMT

As announced on the SharePoint Team blog this morning we released to web three new important updates that affect SharePoint Server 2007, Windows SharePoint Services 3.0, Project Server 2007, Search Server 2008, Search Server 2008 Express and Project Professional 2007.

The Infrastructure Update for Microsoft Office Servers (KB951297) (Download X86, Download X64) is particularly important from a Search and SharePoint Server 2007 perspective as it contains the new Enterprise Search features that were shipped in Search Server 2008 and Search Server 2008 Express that were are not already in SharePoint Server 2007; this includes Federated Search capability, a unified administration dashboard and several Search core platform performance updates.

For an overview of the new federation features please check out this short video which covers how to configure a federated location and configure one of the new federated search Web Parts to show results from that location.

There’s also a growing number of articles on TechNet and MSDN that cover configuring and troubleshooting federation and extending federation with Federated Search Connectors.

The screen capture below shows how federated search results show up on a results page – the results on the right hand side and top left are federated results, the ones at the bottom left are from the local index.

The new Search Administration Dashboard consolidates all of the Search related admin activities into a single place – there’s also some new functionality in the dashboard (There’s greater granularity for content source crawl history and a convenient list that shows currently running crawls and durations for example) and it makes the Search Administrators job much easier by keeping everything close at hand.

The UI looks like the screen capture below:

The update leaves the old Search admin pages intact, the links to them stay in Central Admin (Along with a new link to the new dashboard) so if you’ve made any changes to them or just prefer to use the existing admin pages you’re free to do so.

The other changes are all under the hood and improve Index and Query performance as well as fixing a few bugs. Check out the KB articles below for more details.

Description of the Infrastructure Update for Microsoft Office Servers (KB951297)

Fixes Included in the Infrastructure Update for Microsoft Office Servers (KB953750)

Please read this post on the SharePoint Team blog and the installation instructions thoroughly before you install the Infrastructure Update for Microsoft Office Servers (KB951297) and the Infrastructure Update for Windows SharePoint Services 3.0 (KB951695) on SharePoint Server 2007 or Search Server 2008.

Install the Infrastructure Update for Microsoft Office Servers (Office SharePoint Server 2007)

Install the Infrastructure Update for Microsoft Office Servers (Search Server 2008)

Finally, if you’re wondering why the Infrastructure Update for Microsoft Office Servers (KB951297) and the Infrastructure Update for Windows SharePoint Services 3.0 (KB951695) applies to Search Server 2008 then you’re probably not alone!

There’s a very good reason for both – Search Server 2008 and Search Server 2008 Express are built on the Windows SharePoint Services 3.0 platform (hence the need for the Infrastructure Update for Windows SharePoint Services 3.0 (KB951695)) and secondly the Search features are from SharePoint Server 2007 (hence the need for the Infrastructure Update for Microsoft Office Servers (KB951297)). The latter update includes a few bug fixes since Search Server 2008 and Search Server 2008 Express launched.

We strongly recommend that you install the updates that apply to you as soon as your patching and maintenance schedules permit.

Richard Riley
Senior Technical Product Manager
Microsoft Corp.

New hotfix package for Lotus Notes Protocol Handler

enterprisesearch — Mon, 14 Apr 2008 12:40:10 GMT

If you're working with searching Lotus Notes with SharePoint Server 2007 or Search Server 2008, you may have hit one of the following problems:

"You configure SharePoint Server 2007 to crawl a Lotus Notes content source. In the Manage Content Sources page, you add a new content source by specifying a database name. When you edit the newly created content source, the database name is not listed in the Database Name list."

"You configure SharePoint Server 2007 to crawl a Lotus Notes content source. You first perform a full crawl for the content source. However, when you click Edit content source for the content source, and then you click OK, you have to perform a full crawl again before you can search the content source. This behavior occurs even if you did not change any settings for the content source."

Although a work around is possible by using administration tools like SharePoint Search Admin (http://www.codeplex.com/searchadmin) we recently released a hotfix to fix the issues – KB950280 (http://support.microsoft.com/kb/950280 ), it also solves several other bugs including a fix for the German wordbreaker - if you have problems searching compound words like “Riesentorte” and “Baumgarten”, this hotfix should solve the problem.

Remember, if you have not applied Service Pack 1 on SharePoint Server 2007, please apply it before you install this hotfix. There’s no SP1 requirement for Search Server 2008.

Jie Li
Partner Technology Specialist
Microsoft China

Crawling case-sensitive Web content in SharePoint Server 2007 and Search Server 2008

enterprisesearch — Sat, 26 Jan 2008 09:59:41 GMT

The hotfix for enabling case-sensitive URL's to be crawled with MOSS is pretty well known http://support.microsoft.com/kb/932619 (The fix is also included in MOSS SP1) but what is maybe less well known is that you also need to enable the same registry key in Search Server 2008 to crawl case-sensitive URLs - i.e. it's not switched on by default in MSS. This is also true after installing SP1 for MOSS, you'll need to set the registry key afterwards to enable the fix.

Following the KB article enables case-sensitive crawling, it doesn't enable case-sensitive searching. So for example, MOSS and MSS will be able to successfully crawl and index a URL of http://SERVER/Site/content but a query of "QUERY TERMS" will return the same results as a query of "query terms".

Richard Riley
Senior Technical Product Manager
Microsoft Corp

Microsoft Filter Pack Released to Web

enterprisesearch — Sat, 05 Jan 2008 01:58:02 GMT

Just before the holidays we released a filter pack to the download center that add's support for new Microsoft Office and ZIP file types for several applications that include Microsoft search functionality.

The pack includes IFilters for the following formats: .docx, .docm, .pptx, .pptm, .xlsx, .xlsm, .xlsb, .zip, .one, .vdx, .vsd, .vss, .vst, .vdx, .vsx, and .vtx.

The download is available here; http://www.microsoft.com/downloads/details.aspx?FamilyId=60C92A37-719C-4077-B5C6-CAC34F4227CC&displaylang=en

Links to the installation instructions for each product are here;

Office SharePoint Server 2007 - Office SharePoint Server 2007 registration steps

Search Server 2008 - Search Server 2008 registration steps

SharePoint Portal Server 2003 - SharePoint Portal Server 2003 registration steps

Windows SharePoint Services v3.0 - Windows SharePoint Services v3.0 registration steps

Exchange Server 2007 - Exchange Server 2007 registration steps

SQL Server 2005 - SQL Server 2005 registration steps

SQL Server 2008 - SQL Server 2008 registration steps

There's a slight mistype in the instructions for SharePoint Server 2007 and Search Server 2008 - Details are here http://blogs.msdn.com/ifilter/archive/2008/01/04/errata-for-filter-pack-kb.aspx

Richard Riley
Senior Technical Product Manager
Microsoft Corp

Crawling Sites Protected by Forms or Cookie-based Authentication with Enterprise Search in SharePoint Server 2007

enterprisesearch — Thu, 04 Oct 2007 22:45:00 GMT

Learn how to use Enterprise Search to crawl sites that are protected by forms or cookie-based authentication, and walk through creating a sample custom security trimmer for trimming the returned search results… Read the full MSDN article here

Related resources.

Read the KB article here

Download the required Hotfix here

Download a required tool here