Mike Taghizadeh's Blog : Search

Some Useful MOSS Search Development Related Articles

miketag — Mon, 24 Mar 2008 02:07:00 GMT

Hi all,

I just recently put together an email on some of the articles available for Developers on MOSS Search. So here is my quick list:

Best Practices: Writing SQL Syntax Queries for Relevant Results in Enterprise Search --> http://msdn2.microsoft.com/en-us/library/bb219479.aspx

Creating Search Queries Programmatically by Using the Search Object Model in SharePoint Server 2007 --> http://msdn2.microsoft.com/en-us/library/bb626127.aspx

Creating Search Queries Programmatically by Using the Search Web Service in SharePoint Server 2007 --> http://msdn2.microsoft.com/en-us/library/bb625950.aspx

Evaluating and Customizing Search Relevance in SharePoint Server 2007 --> http://msdn2.microsoft.com/en-us/library/bb499682.aspx

Book Excerpts "Chapter 3: Customizing and Extending the Microsoft Office SharePoint 2007 Search (Part 1 of 2)" --> http://msdn2.microsoft.com/en-us/library/bb608302.aspx

Book Excerpts "Chapter 3: Customizing and Extending the Microsoft Office SharePoint 2007 Search (Part 2 of 2)" --> http://msdn2.microsoft.com/en-us/library/bb608305.aspx

Creating a Custom Search Page and Tabs in the Search Center of SharePoint Server --> http://msdn2.microsoft.com/en-us/library/bb428855.aspx

Creating and Exposing Managed Properties in the Advanced Search Page of SharePoint Server Enterprise Search --> http://msdn2.microsoft.com/en-us/library/bb428648.aspx

Creating and Exposing Search Scopes in SharePoint Server 2007 Enterprise Search --> http://msdn2.microsoft.com/en-us/library/bb428856.aspx

Displaying Search Results in a Grid View in SharePoint Server 2007 --> http://msdn2.microsoft.com/en-us/library/ms497338.aspx

Exposing Enterprise Search in SharePoint Server 2007 by Using Internet Explorer 7 and the Office Research Pane --> http://msdn2.microsoft.com/en-us/library/bb625970.aspx

Creating Custom Enterprise Search Web Parts in SharePoint Server 2007 --> http://msdn2.microsoft.com/en-us/library/bb871647.aspx

Customizing Search Results with Custom XSLTs in SharePoint Server 2007 --> http://msdn2.microsoft.com/en-us/library/bb896018.aspx

Searching in MOSS in the SDK --> http://msdn2.microsoft.com/en-us/library/ms497338.aspx

Class Library reference in MOSS SDK --> http://msdn2.microsoft.com/en-us/library/ms577961.aspx

And others :-)

Mike

Do a Full Crawl after installing MOSS SP1

miketag — Sun, 03 Feb 2008 02:05:00 GMT

Hi all,

I have heard questions around the need to do a Full Crawl, after installing MOSS Service Pack 1. I did not see anything in the documents for SP1 that a full crawl is required or recommended.

But after talking to some of the experts, I was told that a full crawl is recommended but not required. The reason for that is there have been performance improvements made in SP1 that will leverage the information that MOSS collects in the full crawl, to make the incremental crawl much faster. So you do not have to do a full crawl after you move to SP1, and things will be fine ... but you will not see the performance improvements.

Cheers
Mike

Force a Full Crawl for a Site

miketag — Wed, 23 Jan 2008 05:46:00 GMT

Hey all,

So I was doing some work with a customer and I was asked how to do a Full Crawl for a site, without actually doing a Full Crawl and only doing Incremental Crawl. So bascially, they would like to crawl a site fully, using only Incremental crawls. So I was told a quick thing that worked for me in my test. I am not saying that this is the right way, or the recommended way, or the best way to do this, but it is a cool thing that worked for me, so take it with a grain of salt :-)

Go into the site's Site Settings, click the option for Search Visibility to NO, so the site is not available to be searched by MOSS.
Now perform an Incremental Crawl. This site's content will no longer be available in search results.
Now go back into the site and change the setting above to YES.
Now perform an Incremental Crawl again. This Incremental Crawl will now act as the Full Crawl on the site :-)

This is helpful in situations where you can not wait for a Full Crawl but can do few Incremental Crawls during this time. This will do a Full Crawl on the site, even though you are doing an Incremental Crawl. Pretty cool :-)

Give it a try and let me know if it works for you.

Cheers
Mike

Announcing Microsoft Search Server 2008 Express

miketag — Fri, 23 Nov 2007 07:26:00 GMT

Hi all,

I wanted to make sure you saw the following sites:

Announcement on this news at the SharePoint Blog: http://blogs.msdn.com/sharepoint/archive/2007/11/06/announcing-microsoft-search-server-2008-express.aspx

Microsoft Enterprise Search site: http://www.microsoft.com/enterprisesearch/

Microsoft Search Server 2008: http://www.microsoft.com/enterprisesearch/serverproducts/searchserver/default.aspx

Microsoft Search Server Express 2008 [Free]: http://www.microsoft.com/enterprisesearch/serverproducts/searchserverexpress/default.aspx

Cheers
Mike

Customizing and Extending the Microsoft Office SharePoint 2007 Search Excerpts

miketag — Sun, 21 Oct 2007 05:10:00 GMT

Hi all,

I just wanted to make sure that everyone saw these two excerpts from Inside Microsoft Office SharePoint Server 2007 by Patrick Tisseghem, from Microsoft Press

These two excerpts can be found at:

Chapter 3: Customizing and Extending the Microsoft Office SharePoint 2007 Search (Part 1 of 2)

Chapter 3: Customizing and Extending the Microsoft Office SharePoint 2007 Search (Part 2 of 2)

Thanks
Mike

Technical Whitepaper on Microsoft Deployed SharePoint Server 2007 Search

miketag — Mon, 24 Sep 2007 02:00:00 GMT

Hi all,

I just wanted to make sure that everyone saw this technical whitepaper on how Microsoft deployed SharePoint Server 2007 Search internally:

"This document shares the experiences of Microsoft teams in deploying Enterprise Search at Microsoft. Because of the significant amount of knowledge that these teams gained, the experience provides relevant guidance to organizations that want to help improve employee productivity, improve the quality of work produced, reduce duplication of efforts, and take advantage of the financial investment in digital assets by deploying Enterprise Search."

This paper can be found at: http://technet.microsoft.com/en-us/library/bb735129.aspx

Thanks
Mike

Announcing the Microsoft Business Data Catalog Definition Editor for MOSS 2007

miketag — Tue, 28 Aug 2007 20:46:00 GMT

Hi all,

Just want to make sure that you saw the announcement for the new Microsoft Business Data Catalog Definition Editor for MOSS 2007. The BDC Definition Editor tool is now available in the latest version of the MOSS SDK at http://www.microsoft.com/downloads/details.aspx?familyid=6d94e307-67d9-41ac-b2d6-0074d6286fa9&displaylang=en

This tool will make it easy for setting up BDC connections in MOSS, which will allow you to search across your back end applications and databases easier.

For more info, please see http://blogs.msdn.com/sharepoint/archive/2007/08/22/announcing-the-microsoft-business-data-catalog-definition-editor-for-microsoft-office-sharepoint-server-2007.aspx

Thanks
Mike

Recommended Guidelines for Search Boundaries

miketag — Mon, 25 Jun 2007 21:08:00 GMT

Hi all,

I have been asked a lot for guidelines around some of the boundaries for MOSS Search. Here is a quick table that does a good job on this:

Search object	Guidelines for acceptable performance	Notes
Search indexes	One per SSP Maximum of 20 per farm	MOSS supports one content index per SSP. Given that we recommend a maximum of 20 SSPs per farm, a maximum of 20 content indexes is supported. Note that an SSP can be associated with only one index server and one content index. However, an index server can be associated with multiple SSPs and have a content index for each SSP.
Indexed documents	50,000,000 per content index	MOSS supports 50 million documents per index server.
Content sources	500 per SSP	This is a hard limit enforced by the system.
Start Addresses	500 per content source	This is a hard limit enforced by the system.
Alerts	1,000,000 per SSP	This is the tested limit.
Scopes	200 per site	This is a recommended limit per site. We recommend a maximum of 100 scope rules per scope.
Display groups	25 per site	These are used for a grouped display of scopes through the user interface.
Crawl rules	10,000 per SSP	We recommend a maximum 10,000 crawl rules irrespective of type.
Keywords	15,000 per site	We recommend a maximum of 10 Best Bets and five synonyms per keyword.
Crawled properties	500,000 per SSP	These are properties that are discovered during a crawl.
Managed properties	100,000 per SSP	These are properties used by the search system in queries. Crawled properties are mapped to managed properties. We recommend a maximum of 100 mappings per managed property.
Authoritative pages	200 per relevance level	This is the maximum number of sites in each of the four relevance levels.
Results removal	100	This is the maximum recommended number of URLs that should be removed from the system in one operation.
Crawl logs	50,000,000	Number of individual log entries in the crawl log.

For more information, see the "Plan for Software Boundaries" at: http://technet2.microsoft.com/Office/en-us/library/6a13cd9f-4b44-40d6-85aa-c70a8e5c34fe1033.mspx?mfr=true

Hope that helps
Mike

Configure MOSS to crawl Lotus Notes

miketag — Wed, 06 Jun 2007 04:26:00 GMT

Hi all,

I have been asked about how MOSS can be configured to crawl Lotus Notes. Below are some links that may be useful:

Configure MOSS 2007 to crawl Lotus Notes:
http://technet2.microsoft.com/Office/en-us/library/82c7c354-6347-4ae8-b5f8-7d0cdfe432401033.mspx

Configuring the Lotus Notes Protocol Handler:

http://office.microsoft.com/en-us/sharepointportaladmin/HA011603581033.aspx

Information about installing the Lotus Notes Protocol Handler for SharePoint Portal Server 2003:
http://support.microsoft.com/default.aspx/kb/830971

Secure crawls of Lotus Notes with SharePoint:

http://blogs.msdn.com/edhild/articles/473060.aspx

Search and Index Lotus Notes:
http://www.sharepointblogs.com/helloitsliam/archive/2007/01/09/17654.aspx

Displaying Correct Titles of Lotus Notes Documents in SharePoint Search Results:

http://meiyinglim.blogspot.com/2007/01/displaying-correct-titles-of-lotus.html

Use MOSS 2007 to index a Lotus Notes Database:

http://meiyinglim.blogspot.com/2007/01/using-sharepoint-2007-to-index-lotus.html

How to Configure Search to honor Lotus Notes Security Settings:

http://support.microsoft.com/default.aspx?scid=kb;EN-US;Q288816

Hope it helps

Mike

SQL Syntax Queries in MOSS Search

miketag — Fri, 11 May 2007 04:54:00 GMT

Hi all,

I have been asked few times for best practices on writing SQL Syntax Queries with respect to MOSS Search. There are great articles in the MOSS SDK and MSDN on best practices. In case folks are not aware of these, here are couple of quick links:

Best Practices: Writing SQL Syntax Queries for Relevant Results in Enterprise Search: http://msdn2.microsoft.com/en-us/library/bb219479.aspx

Enterprise Search SQL Syntax Reference: http://msdn2.microsoft.com/en-us/library/ms493660.aspx

Hope that helps
Mike

Planning your MOSS Search Team

miketag — Sun, 25 Mar 2007 03:13:00 GMT

Hi all,

I have been asked many times by other consultants or customers, how to plan the MOSS Search team. Before planning and rolling out the deployment of MOSS, you need to know what your Search planning and Operations team might need to look like.

There are guidelines on Microsoft's TechNet site on what the Search planning team might look like, what the Search Process might include and plan your Search Operations and Deployment team. You can find this information at: http://technet2.microsoft.com/Office/en-us/library/b4b9ceae-3837-4a79-8097-d6381500e4401033.mspx

There is also a handy worksheet for customers to track who their Search administrators are for a MOSS deployment and their details such as their name, user or group account, names of SSPs they will administrator and other useful information. I recommend you download this worksheet and use it for your MOSS Search deployment. You can find this at http://go.microsoft.com/fwlink/?LinkId=73621&clcid=0x409

Hope that helps
Mike

MOSS Search Performance and Capacity Whitepaper Released

miketag — Wed, 28 Feb 2007 20:23:00 GMT

Hi all,

The MOSS Search Performance and Capacity whitepaper is now available at TechNet at: http://technet2.microsoft.com/Office/en-us/library/5465aa2b-aec3-4b87-bce0-8601ff20615e1033.mspx?mfr=true

This document dives into performance and capacity scenarios, discussing lab testing and topology and user profiles. It then dives into recommendations for server sizing and hardware and many other Search related areas.

Thanks
Mike

Did You Mean?

miketag — Mon, 12 Feb 2007 04:37:00 GMT

The Did You Mean? feature is used by the query engine to catch possible spelling errors and to provide suggestions for queries. The Did You Mean? feature builds suggestions by using three components:

Query log: Information tracked in the query log includes the query terms used, when the search results were returned for search queries, and the pages that were viewed from search results. This search usage data helps you understand how people are using search and what information they are seeking. You can use this data to help determine how to improve the search experience for users.
Dictionary lexicon: A dictionary of most-used lexicons provided at installation time.
Custom lexicon: A collection of the most frequently occurring words in the corpus, built at query time by the query engine from indexed information.

The Did You Mean? suggestions are available only for English, French, German, and Spanish.

The information above was taken from the whitepaper called "Plan for building multilingual solutions" available at http://technet2.microsoft.com/Office/en-us/library/87065c9d-d39d-479d-909b-02160ec6d7791033.mspx?mfr=true

Thanks
Mike

MOSS Search - Customer Proof of Concept Scoping and Typical Tasks

miketag — Thu, 18 Jan 2007 19:57:00 GMT

Hi all,

There are a lot of customers today that want to engage in a Proof of Concept or Pilot with MOSS 2007 Search. These customers would like to setup MOSS Search, configure it, index and crawl all their content and then present the results to their end users. The great thing about MOSS Search is that it is easy to setup MOSS and configure the Search to allow customers to see how this works in their environment with their content.

So I get pulled in a lot to scope out Pilots or Proof of Concepts for customers around MOSS 2007 Search. Below, I have included a sample on a typical MOSS 2007 Search POC. At the beginning, I have mentioned some of the areas that I like to investigate before starting to scope the POC. These questions can be asked from the customer before the POC starts. Once you have these answers, then those answers will then decide how long the POC might last, what work needs to be done, what the success factors are and what the tasks are in each week.

Hope that helps,
Mike

Discussion points for the pre-POC call:

What is their current SharePoint or CMS infrastructure/deployment today? Do they want to search any of these systems?
What is their Authentication/Authorization requirements for this POC?
Do they have any database content that they want to search in MOSS?
What type of hardware, network, load balancing system do they have that can be used for the POC?
Can we install Office 2007 Client bits on some client hardware for the POC?
Where is your majority of content? Web sites? SharePoint sites? File system? Exchange folders? Lotus Notes? Other areas?
What are the type of content you have? Office Documents? PDFs? Images? Videos? Others?
Do they want to index and search any back end applications?
Where else can we point MOSS to, for indexing and searching of content? SAP? Siebel? PeopleSoft? ASP.Net applications? Other databases?
What is the success factor for this POC?
What are the requirements around people search?
Any already existing search products today?
Any requirements around presentation of results to end users?
Other questions that will determine the tasks for the POC

Here is a sample MOSS Search POC task breakdown:

Installing MOSS and Office 2007
Setting up SSP, Web Applications, and Site Collections
Setting up Authentication and Authorization
Importing User Profiles into MOSS for index/searching
Building Site Collections and sites and assigning users to these
Indexing HTTP/SharePoint content in MOSS
Setting up the Search site to return results
Configuring Search web parts and other search related areas
Setting up and creating BDC definitions for external content to be searched such as SAP or Siebel
Configuring search to include BDC data

Customize the Advanced Web Part and/or add People Search to MOSS
Build MOSS master pages, content types and page layouts
Create styles, import images and build the look and feel of MOSS
Have BDC and People Search configured and working
Setup Keywords for the content
Setup up Best Bets
Modify the synonyms and thesaurus for Search

Write web parts for custom queries and custom searches
Write other web parts as needed
Work on relevancy of content
Setup searching of other data such as PDF
Customize search reports
Improve search performance
Build XML and XSLT for custom results
Other MOSS related areas
POC Closure and Recap

MOSS Search Word Stemming - Part 2

miketag — Wed, 27 Dec 2006 09:35:00 GMT

So how Does MOSS Expand Search Query Terms to Related Words?

Here is how this works in MOSS:

In MOSS, stemming is used in combination with the word breaker component which determines where word boundaries are. The word breaker is used at both index and query time while the stemmer is used only at query time for most languages (the exceptions currently are Arabic and Hebrew) to perform both morphological analysis and morphological generation. In the case of Arabic and Hebrew, stemming is restricted to morphological analysis at both query and index time. A stemmer links word forms to their base form. For example, ”running,” ”ran,” and ”runs“ are all variants of the verb ”to run.” Stemming is currently turned off by default for some languages including English. Stemmers are only available for languages which have significant morphological variation among their word forms. This means that for languages where stemmers are not available (such as Vietnamese) turning on this feature in the Search Result Page (CoreResult Web Part) will not have any effect, since in such languages exact match is all that is needed.

Word Stemming is NOT the same thing as Wild Card Searching, which our engine supports as well. Wild Card searching has to do with doing searches with * in the query. This means you are asking the search engine to find you all words that start with the text string and end with anything, since * means match any character any number of times until you reach the end of the word which in most languages (excluding most East Asian languages) is indicated by a white space. So a search query using * such as "Share*" will return results including "SharePoint", while a search query using morphological processing would bring back "sharing", which is an inflectional variant of Share. Wild Card searching and Word Stemming are often used to refer to the same thing but they are in fact separate and different mechanisms which can return different results.

Word Stemming would bring back words closely related to the query terms (usually inflectional variants for most languages, but for some languages derivational variants as well).

For example, for the following queries, here are some sample results

If you type in "run" --> in addition to exact matches on “run”, it will bring back matches on "runs", "ran" and "running"
If you type in "page" --> in addition to exact matches on “page” it will bring back matches on "pages", "paged" and "paging"
If you type in "basket" --> in addition to exact matches on “basket” it will find "baskets", but it will not find "basketball". A wild card search for “basket*” would find basketball, which our engine supports and I will discuss this in another article. Word Stemming does not handle this currently because we have focused on matching inflectional variants of words only rather than derivational variants.

However this option is turned off by default out of the box for English and some other languages. You can turn this on by going to the Search Results Web Part, and then Options and turn on this feature which is called “Enable Search Term Stemming”.

Thanks for Ian Johnson from the Natural Language Group at Microsoft for providing his feedback on this.

Hope that helps

Mike