Mike Taghizadeh's Blog : SharePoint

MOSS Search Word Stemming - Part 1

miketag — Thu, 21 Dec 2006 21:37:00 GMT

Hi all,

So I have been getting some questions around how MOSS does Word Stemming, so here is what I have been able to gather/research:

MOSS Search Stemming:

The word Stemming refers to the process of stripping off endings of words at query/index time so that different search terms will match and retrieve documents containing related words in the index. A hypothetical example of this would be the reduction of the related words “diet”, “diets”, “dieting”, “dieted”, “dietary”, “dietician”, “dieticians”, etc to a single stem “diet” which would allow a query for any one of these words to be matched against documents containing any of the other words. Computational implementations of stemming are typically not based on linguistic notions of stem and affix, but rather just frequently occurring character sequences.

The correct linguistic terminology for the process of stemming described above is morphological processing, meaning the linking of a given word form to its base form and other related word forms.

Morphological processing has two main aspects:

1. Morphological Analysis

2. Morphological Generation.

Morphological Analysis refers to the process of analyzing a given word form by assigning its correct morphosyntactic data (such as person, number, gender, tense, etc) and identifying its internal structure in terms of base form and any prefixes and suffixes which can be inflectional (and do not change the part of speech or meaning of the word: e.g. diet, diets) or derivational (which do change the part of speech or meaning of the word, e.g. diet, dietary).

Morphological Generation goes ones step further. In addition to analyzing an inflected form down to its base form, it also generates all inflected forms which are related to the same base form. In this case, the inflected verb form “loved” is reduced to the base form “love” by Morphological Analysis and then Morphological Generation generates all related inflected forms, i.e. “love”, “loves”, “loving” “loved” which are then matched against the document index and all matches are retrieved and displayed to the user (normally in search engines exact matches are given a higher weighting than the related words and are displayed at the top of the results list).

The decision as to whether to apply Morphological Analysis or both Morphological Analysis and Morphological Generation in Search depends on two major considerations: (a) whether there are any restrictions on the size of the search index, and (b) the level of morphological complexity of the language.

(a) If we do not want the size of the index to increase as a result of this morphological processing, then we will leave the index unchanged and use Morphological Analysis, and Morphological Generation at query time to expand a given search query term into a list of inflectionally- and/or derivationally-related word forms which are all then matched against the search index and return a list of results in which exact matches are listed first. If the size of the index is permitted to increase then we can use Morphological Analysis at index time to store a base form along with each inflected form in the index. At query time we will also use Morphological Analysis to attach a base form to the query term and then search on both items. All matches for both query term and its base form will be retrieved and any exact matches on the query term listed first.

(b) If the morphology of a particular language is very rich then a very large number of query terms could be generated at run time. This could have a profound effect on processing efficiency. In languages such as Arabic and Hebrew therefore which have a very large number of forms of a single word (the number can get into the thousands), it is therefore preferable to avoid morphological generation and just use Morphological Analysis at both index and query time. This increases the size of the index but has the advantage that the number of forms of a word stored in the index and generated from the query will not exceed 2 (the word itself and its associated base form).

PS: For Arabic related issues and questions, please see:

                        Arabic specific MOSS http://www.microsoft.com/middleeast/arabicdev/office/office2007/SharePoint.aspx ,
                        Arabic MOSS search http://www.microsoft.com/middleeast/arabicdev/office/office2007/Search.aspx ,
                        Arabic 2007 office servers http://www.microsoft.com/middleeast/arabicdev/office/office2007/office2007server.aspx

The key point of difference between wild card searching and word stemming (or morphological processing) is that the former is just string based and allows the user to find in some cases both inflectional and derivational variants of the query term. The Stemming approach deals very well with inflectional variants but we currently don’t handle derivational morphology for most languages. In languages where we do have some limited treatment of derivational morphology such as Arabic and Hebrew, this treatment is limited to high frequency terms.

Thanks for Ian Johnson from the Natural Language Group at Microsoft for providing his feedback on this.

Stay tuned for Part 2, where I will explain how MOSS does Stemming in more detail.

Hope that helps

Mike

Differences Between MOSS Content Sources

miketag — Wed, 06 Dec 2006 21:08:00 GMT

I have been getting this question few times now with some of my MOSS Search customers. I have been asked what is the difference in setting up a "SharePoint Sites" MOSS Content Source versus a "Web Sites" MOSS Content Source.

Here I have tried to add some of my thoughts on these two:

Both these Content Sources allow you to have Name to describe the Content Source, so that you know the name for tracking and other Search related tasks
They both allow you to have Start Addresses for crawling this Content Source. In the Web Sites Content Course, this can include any content, from a single web page to a whole entire web site. For a SharePoint Sites Content Source, this can include Office SharePoint Server sites and WSS Sites.
For setting the Crawl Settings, this is where the difference between the two come about. In a Web Sites Content Source, you can specify that you only want to crawl the server of which you entered the Start Address above, or only crawl the first page of the start address above, or .... (which is my favorite) ... have a custom crawl settings set up. Here you can specify Server Hops and Page Depths. These two options are not available in a SharePoint Sites Content Source.
Page Depths are the number of links to follow on the same hostname. So for your SharePoint Sites Content Source, if you have a Page Depth of 1, the crawler will follow links from the home page and then stop.
Server Hops are the number of host name changes that the crawler will make. For example, if you have a Server hop of 1, a link on your site will be followed to any other host name, but it will not be followed to another server hop.
One additional difference is that the SharePoint content source allows users to crawl a single WSS site collection, which is not possible in a Web Sites content source. Meaning, if you want to crawl only a site collection, you have to put its URL in a SharePoint content source like http://myserver.com/sites/mikesitecollection and select the radio button to “Crawl only the SharePoint site of each start address”. If you put the same start address in a Web Sites content source, it will go all the way to the top (http://myserver.com) and start crawling because that is the default for all SharePoint content.
Also the Web Sites Content Source can figure out that the starting address is a SharePoint site from the response header during the crawl, and then switch protocol handlers for crawling.
Of course, both content sources allow Full and Incremental Crawls and they both allow you to create schedules.

Hope that helps.

Thanks
Mike

Feature Comparisons in SharePoint Search Versions

miketag — Mon, 04 Dec 2006 09:11:00 GMT

I have been asked this question few times by my MOSS Search customers. Question being: What is the difference in features in WSS 3.0, MOSS 2007 and Office SharePoint Server 2007 for Search?

One thing to note is that the Search in the 3 products are all based on the same Microsoft Search core indexing engine. However, there are differences between the 3 SharePoint search versions. This table does a comparison of features in the 3 products:

Feature	Search in WSS 3.0	Office SharePoint Server 2007 for Search	Enterprise Search in MOSS 2007
What can be indexed	Local SharePoint content	SharePoint sites, Microsoft Exchange Server content, file shares, Lotus notes, custom content.	SharePoint sites, Microsoft Exchange Server content, file shares, Lotus notes, custom content, line-of-business (LOB) content.
Rich, relevant results	Yes	Yes	Yes
Alerts RSS Did you mean? Collapsing of duplicates	Yes to all	Yes to all	Yes to all
Best bets Results removal, Query reports	No to all	Yes to all	Yes to all
Search Center / Tabs	No	Search Center without Tabs	Search Center with Tabs
People search Knowledge network	No to all	Yes to all	Yes to all
Business Data Search	No	No	Yes
Query Web service	http://<site>/_vti_bin/spsearch.asmx	http://<site>/_vti_bin/search.asmx	http://<site>/_vti_bin/search.asmx
Security trimming of search results	Yes; supports default security trimming only	Yes; supports default security trimming only	Yes; supports default and custom security trimming
Query syntax	Windows SharePoint Services 3.0 Search SQL Syntax Windows SharePoint Services 3.0 Search Keyword Syntax URL syntax	Enterprise Search SQL Syntax Reference Enterprise Search Keyword Syntax Reference Enterprise Search URL Syntax Reference	Enterprise Search SQL Syntax Reference Enterprise Search Keyword Syntax Reference Enterprise Search URL Syntax Reference

For more information, please see this article in the MOSS SDK: http://msdn2.microsoft.com/en-us/library/aa981559.aspx

Hope that helps,
Mike

Tool to export and import MOSS Search Configurations from one SSP to another

miketag — Mon, 04 Dec 2006 06:54:00 GMT

I just wanted to let the readers know of a tool that allows taking MOSS Search settings from one SSP to another. This tool written by Angus Logan, can take the output of Search settings from one MOSS SSP and import them into another SSP. This eliminates errors and saves time when deploying Search settings from one environment to another. Such as Development --> Staging/QA --> Production.

Check it out at: http://blogs.msdn.com/angus_logan/archive/2006/11/16/tool-to-export-and-import-search-configurations-from-an-ssp-in-moss.aspx

Hope that helps
Mike

Finding out the Crawl Status of a piece of content in MOSS Search

miketag — Fri, 01 Dec 2006 10:10:00 GMT

There is a great feature in MOSS Search that some folks may not know about. This is regarding finding out the status of an item in MOSS Search. For example, if you want to know if a site, a document or a link was crawled or not by MOSS Search, we have the feature called Crawl Logs.

From the SharePoint Central Administration site, go to the SSP site. Click on Search Settings under Search, and then Crawl Logs. Before you go any deeper, there is a text box that allows you to enter any URLs that begin with the hostname/path that you want to enter, and it will give you the crawled status. See first screen shot below. However, this gets much more interesting, detailed and useful if you go deeper.

Now click on any Host Name you have there under Crawled Content Status. Here you can find out the status of any item. You can filter on time/date, Content Sources, Status Type (Success, Warning or Error) and other criteria. This page tracks information about the status of crawled content in MOSS and can be very useful when you are trying to figure out if your content was crawled. See second screen shot below.

For more info on this great feature, see the MOSS SDK.

MOSS Search - Some Microsoft Links

miketag — Fri, 01 Dec 2006 09:21:00 GMT

I have been compiling links to current documents/links/articles on Mcirosoft related sites regarding Microsoft Office SharePoint Server 2007 Search. I am sure there are more links out there, but this is a good list to start with:

MOSS 2007 Search Overview in SDK: http://msdn2.microsoft.com/en-us/library/ms495418.aspx
Searching in MOSS in SDK: http://msdn2.microsoft.com/en-us/library/ms497338.aspx
Configure MOSS Search Service at TechNet: http://technet2.microsoft.com/Office/en-us/library/b28ba53d-a3e8-440f-9fcb-f592d858894a1033.mspx?mfr=true
Plan for Performance and Capacity in MOSS at TechNet: http://technet2.microsoft.com/Office/en-us/library/eb2493e8-e498-462a-ab5d-1b779529dc471033.mspx?mfr=true
Plan MOSS Search at TechNet: http://technet2.microsoft.com/Office/en-us/library/2f237fcb-d8c5-4ccc-943b-25fb361f294d1033.mspx?mfr=true
Security Considerations for Search in MOSS at TechNet: http://technet2.microsoft.com/Office/en-us/library/b28ba53d-a3e8-440f-9fcb-f592d858894a1033.mspx?mfr=true
TechNet Webcast: In Depth Look at MOSS Search: TechNet Webcast- An In-Depth Look at SharePoint Server 2007 Search Technology (Level 200)

Hope that helps
Mike

Microsoft Enterprise Search

miketag — Fri, 01 Dec 2006 09:04:00 GMT

I just wanted to make sure that folks are aware of some blog postings and links/whitepapers that Microsoft has today regarding Enterprise Search:

Start with this article by Arpan Shah on Enterprise Search: http://blogs.msdn.com/arpans/archive/2005/11/15/492946.aspx
Then read this article by him on Why choose Microsoft for Enterprise Search: http://blogs.msdn.com/arpans/archive/2006/02/22/537653.aspx
Then read his next article on Microsoft Enterprise Search 101: http://blogs.msdn.com/arpans/archive/2006/07/29/682715.aspx

Also take a look at Microsoft's Enterprise Search page at: http://www.microsoft.com/livesearch/default.mspx

Also take a look at Windows Desktop Search home page at: http://www.microsoft.com/windows/desktopsearch/default.mspx

Also look at Microsoft Office SharePoint Server 2007 at: http://office.microsoft.com/en-us/sharepointserver/default.aspx

There is also a great whitepaper at: http://download.microsoft.com/download/6/b/3/6b303b61-dbf9-4d20-907c-c8d7213d112b/May%202006%20Enterprise%20Search%20Whitepaper.doc

Hope that helps
Mike