Jie Li's GeekWorld : Microsoft Search Server 2008

PDF iFilter Battle, second round

Jie Li — Mon, 09 Mar 2009 22:30:34 GMT

If you still remember the last round of our PDF iFilter battle, FoxIT won it. Now in this round, we bring in another challenger: TET PDF iFIlter. It is also avaliable on x86 and x64, free for non-commercial desktop use, will need a license for Server installation.

So here's the new result for file set II:

	File Number	Total File Size(MB)	Avg File Size(MB)	Crawl Time(m:s)	Crawl Time(s)	File Per Second	Success	Error
FoxIT	2676	2406	0.90	7:46	466	5.74	2759	0
Adobe	2676	2406	0.90	40:58	2458	1.09	2757	2
TET	2676	2406	0.90	13:48	828	3.23	2752	0

I also obtained an archive copy from People's Daily, from 2001 to 2006. ~20,000 PDF files, 13.4GB total. Tested on a 8 cores XEON box.

	File Number	Total File Size(MB)	Avg File Size(MB)	Crawl Time(h:m:s)	Crawl Time(s)	File Per Second	Success	Error
FoxIT	19890	13793	0.69	00:30:53	1853	10.73	19884	7
Adobe	19890	13793	0.69	05:19:04	19144	1.03	19887	4
TET	19890	13793	0.69	01:40:09	6009	3.31	19879	12

And licensing comparsion for production(USD):

	Desktop	Server	1-2 Cores Per Server	4 Cores Per Server	8+ Cores Per Server
Adobe	Free	Free	Free	Free	Free
Foxit	Free	Not Free	329.99	589.97	1109.93
TET	$119 for commercial usage	Not Free	595	595	595

Summary

It is good to see another vendor joined this market. TET showed good performance, although still behind Foxit. But it's licensed based on servers not cores, the cost would be lower than Foxit if you have a typical 2 way quad cores box.

PDF iFilter Battle! FoxIT vs.. Adobe, 64bit version

Jie Li — Wed, 10 Dec 2008 07:46:28 GMT

After so long a time Adobe finally released its 64bit version of PDF iFilter!

http://www.adobe.com/support/downloads/detail.jsp?ftpID=4025

“In response to customer requests, Adobe is releasing Adobe PDF iFilter 9 for 64-bit platforms, which will allow searching PDF files on Microsoft® Windows® 64-bit platforms for applications such as Microsoft Office SharePoint Server 2007, Microsoft Exchange Server 2007, and Microsoft SQL Server 2005.”

But, what about performance? How does it compare with FoxIT 64bit PDF iFilter?

My friend Deb Haldar did a performance test last year for their 32bit iFilters. You can find the result here: FOXIT vs.. Adobe PDF IFilter [ 32-bit only ]. Let’s say, FoxIT 32bit PDF iFilter is more than 4 times faster than the Adobe one.

Will the story change in 64bit age?

I picked about two sets of PDF files. Set I contains ~1000 PDF files, 1.7 GB in total. Set 2 contians ~2600 files, 2.4G in total. Language is mixed by 30% Chinese, 70% US English. The hardware spec is a two-way dual core XEON at 3.4GHz, 4G Ram. SharePoint was patched with October CU. Here’s the result.

	File Set	File Number	Total File Size(MB)	Avg File Size(MB)	Crawl Time(m:s)	Crawl Time(s)	File Per Second	Success	Error
FoxIT	I	1041	1751	1.68	6:02	362	2.88	1064	0
Adobe	I	1041	1751	1.68	30:03	1803	0.58	1063	1
FoxIT	II	2676	2406	0.90	7:46	466	5.74	2759	0
Adobe	II	2676	2406	0.90	40:58	2458	1.09	2757	2

On average, FoxIT x64 PDF ifilter is still ~5 times faster than the Adobe one. But FoxIT charges 330 USD for a 2 core machine, while Adobe PDF iFilter is free. So if PDF indexing is the key to your business, go with FoxIT to get much better performance. If not, you may play with Adobe PDF iFilter to furfill some simple and basic request.

Search Suggestions in IE8 with SharePoint/Search Server

Jie Li — Fri, 05 Sep 2008 03:23:13 GMT

IE8 Beta 2 has been there for a while. Although it is not supported to use together with SharePoint products yet (there’s no chance to support a beta product), you can still try it out. There’re couple of new features introduced like WebSlices, Accelerators and Search Suggestions.

Search Suggestions! Isn’t it cool to make your intranet SharePoint portal to be a Search Provider and have this lovely suggestion feature?

However, by default you can make SharePoint a Search Provider, but no way to add a suggestion feature.

So here it comes – an update to “Search As Your Type(SAYT)” codeplex project, with Search Suggestion working in IE8! And I also included a small green “S” logo icon file for that, all free:)

Since it needs time to update codeplex project, I’ll put something here.

1. Install SAYT on your SharePoint Server/Search Server as instructed. Do some test searches, to make sure it works.

2. Download new update from here:

http://cid-8007edf5c56fc334.skydrive.live.com/self.aspx/Public/SharePointSearchIE8.zip

3. Extract this zip file, copy all files to the directory you put GetInfo.aspx in first step, and overwrite it.

4. Modify ssprovider.xml as needed. Replace SharePointSearchCenter and SAYTUrl with your own ones.

5. Use IE8 to navigate to add.html, and add search provider.

6. Choose the green “S” provider and try it out!

You can add a sample provider on http://www.mssearch.cn:8099/add.aspx, and try type in “search server” to see the result.

Disclaimer: This code is not support by Microsoft, if you have problems, leave your comment here. SAYT Codeplex project will be updated later to include this feature.

“Evil” way to federate search results through a password protected proxy

Jie Li — Thu, 24 Apr 2008 03:11:00 GMT

In real world environment, people sometimes use password protected proxy to make company employees to access the Internet. Most of the time, that is a basic authentication. So in this kind of environment, the federated search webpart of Microsoft Search Server 2008 will not work out-of-the-box because we only support non-password protected proxy.

But is there any way to workaround?

Yes, otherwise why I’m talking about it?

For the word “evil”, I’m not referring to the definition of the word from a “not evil” company. My “evil” is always some kind of tricks, or hacks, and you will love them because they can really solve problems. BTW – in MMO RPGs I’m always a chaotic neutral character, but I like evil ones - that’s my best description.

The theory is to make a data tunnel through the password protected proxy, so we can map external website to local port, and federate the search result. There’re some applications which can do the job, but here we will use HTTPort as an example.

Here’re the steps!

1. Get a copy of HTTPort from www.htthost.com, the newest version is HTTPort 3.SNFM. Install it.

2. In proxy configuration window, fill in your proxy server name and port, check “Proxy requires authentication” and then input your username and password for accessing this proxy.

3. Check the RSS feed website domain name you want to federate. In this example, we are using Live Search China. The domain name is “cnweb.search.live.com”.

4. Click Port mapping tab of HTTPort, and add a new port tunnel. Fill in a local port, for example, 991, then fill in remote host name and port.

5. Switch back to proxy tab. press start button in lower right corner.

6. Check if it works in your browser: replace domain name of your RSS feed with 127.0.0.1:991. If everything is going on well, and you are lucky enough, the RSS feed will be there and you can make Search Server federate it through this new local URL!

7. Just some note: Not every service can be federated like this. If the target website has more security check, for example Yahoo search, the RSS feed cannot be fetched through such tunnel. Therefore you have to consider other ways, or spend some time to imporve this evil hack:).

HTTPort is a free software written by Dmitry Dvoinikov.

What may happen when I crawl MILLIONS of files in MOSS/MSS? Part II - Why I need X64 instead of X86?

Jie Li — Sun, 13 Apr 2008 20:44:35 GMT

In last post of this series we talked about the crawl time and and CPU usage. This time we will talk about process memory usage and x86/x64 issue.

Many customers asked me questions about x86/x64 comparison. Most of them consider x64 would be a benefit, but they don't know what kind of benefit it really brings.

So now I tell you why it's needed.

As we know, SharePoint/search server has three layers. Each layer can only be single architecture.

Layer 1: Web Front Server (WFE)

WFE is needed to host the web site, process different user events. This layer, of course can be and should be x64, unless you have some pretty old webpart which still need x86. At this layer, the most memory consummation comes from IIS(w3wp.exe). For IIS in X86, it can only use at about 1.1-1.2G memory. If you hit this barrier, the process may just hang there. This situation happens when very big number of request lasted for a long time(several hours or days, it depends on how many users are accessing the site at the same time). You can have multiple WFE in one server farm.

It's quite important to have IIS recycle automatically, and sometimes you even need to manually recycle it.

Layer 2: Query Server.

Query Server hosts query engine. It continually receive index propaganda from index server. When users make a query, it will be sent to query engine. Query engine will check SQL Server for document properties, and check index for content chunk. So the disk performance is important for query server when query load is high. Other workload, like query time security trimming by custom security trimmer, is also done by Query Servers.

You can have multiple Query Server per farm. And Query Servers, should be x64 if you have the hardware.

Layer 3: Index Server.

Index Server is the spider. Because you can only have one Index Server currently in SharePoint 2007/Search Server 2008, you need to take great care of it. DO NOT put ANY other applications on it, DO NOT share the box with SQL. If you do, you will quickly be hit by the bad performance.

Yes, Index Server should be x64. The problem is, sometimes we cannot make it x64. For example, we have a 32bit ifilter which does not have any x64 implementation but is very important to customer business, or we only have 32bit protocol handler like Lotus Notes PH... In these situations, you can only have a 32bit Indexer.

You may suffer from the same limitation when using 32bit Indexer, especially in Lotus Notes. When you continues crawl too many Notes docs into SharePoint, index engine may hang there for several hours because of memory limit. I suggest, if you happen to come across such problem, make a simple application to monitor and automatically restart search service after one DB is finished. This can manually recycle the memory used by the engine.

In the future, nearly everything can be made x64. So stay tuned.

Database: SQL Server

No matter querying or indexing, the backend database is always a main contributor to the whole performance. So make it independent, make it faster, and even make it cluster. This will help with overall performance. x64 is the best choice for DB.

Index FTP content with SharePoint 2007/Search Server 2008

Jie Li — Thu, 10 Apr 2008 10:03:00 GMT

Question: Can SharePoint 2007/Search Server 2008 index and search ftp?

Answer: OOTB you cannot. But we have a FTP protocol handler in SharePoint 2001 ResKit, and that one can be used on SharePoint 2007 or Search Server 2008.

Important Notice: This approach is NOT supported by Microsoft. It is just for your test purpose only, and should not be used in an important production envoriment.

Some restrictions may apply: x86 and anonymous ftp only.

Here's the steps.

1. Get a copy of ftpph.dll. It is in SharePoint 2001 ResKit.

2. Copy it to c:\Program Files\Microsoft Office Servers\12.0\Bin

3. Register this dll. Open command prompt, navigate to c:\Program Files\Microsoft Office Servers\12.0\Bin, run "regsvr32 ftpph.dll".

4. Open Regedit, navigate to HKLM\SOFTWARE\Microsoft\Office Server\12.0\Search\Setup\Protocol Handlers

5. Insert a new string item: "ftp". The value is "FtpPH.SearchProtocol.1".

6. In command prompt, restart search service by "net stop osearch" and "net start osearch"

7. Download SharePoint Search Admin at http://www.codeplex.com/SearchAdmin , and then add a new custom content source for your ftp site.

8. Start full crawl, and you will see the crawl log. Job done!

Search Community Toolkit on Codeplex

Jie Li — Thu, 13 Mar 2008 04:21:58 GMT

http://www.codeplex.com/sct

Richard Riley (3x to your xbox racing wheel:)) announced the new Community Toolkit project on Codeplex, which gives you a list of opensource enhancement for Search Server 2008 and SharePoint 2007.

http://blogs.msdn.com/enterprisesearch/archive/2008/03/12/search-community-toolkit.aspx

Yes, Smart Search, Search as you type, Faceted Search, Search Admin... are all there. And there're two new additions: Search Asp.net Server Controls for your asp.net applications, and Search Center Branding Tool which can kick the boring sharepoint interface into hell by just a few modification:)

It's time to try for yourself!

Smart Search for SharePoint v1.2

Jie Li — Fri, 07 Mar 2008 21:00:06 GMT

Many many thanks to Shawn Feldman!

Quote from him:

"I was having a few issues getting smart search installed and granting it the right permissions to access the db (I’m running a multi account environment) so I put it in a feature, Gac’d it, and added SPSecurity.RunWithElevated for whenever it makes a database connection. Here’s the code if you’re interested."

It's my fault. I know this problem when I was working on a Internet(anonymous) facing SharePoint site. The problem of SmartSearch v1.1 is it will use the current user to access db, of course, cannot be successful when using IUSR_XXXX accounts. RunWithElevated is the definitely the solution but I don't have time to do as I'm alwasys travelling around.

So here you are:

Smart Search for SharePoint ver 1.2 - Mar 8 2008

I think this is the reason why we made these open source tools - everybody can contribute to it, and make our work better.

Build Custom Federated Search Connector in Microsoft Search Server (and SharePoint) - Solve Problems and Extend Your Ideas

Jie Li — Fri, 29 Feb 2008 14:17:23 GMT

I assume the read of this article understand what is federated search. So we already know that in order to use Federated Search webpart in Search Server, you need to provide a RSS feed to it, which can also be called "OpenSearch" stuff.

But, not every application you search will return this kind of RSS/ATOM feed. For example, Google, Baidu and many other web sites. So how can you federate search results from this kind of web sites?

http://msdn2.microsoft.com/en-us/library/bb931083.aspx

Scenario 2: Connecting to an External Search Site That Returns Results in HTML Format

Scenario background: The site is configured to use Anonymous access.

Possible solution: Use a Web application outside of the context of a SharePoint site, which contains a lightweight ASPX page that does the following:

Submits a search request to the site by using the search terms passed in the initial request URL.

Converts the results in the HTML response received from the external search site to RSS format.

Returns the RSS XML in the response to the search server.

In this scenario, the federated connector’s Web application could be located on a remote server; however, a simpler solution is to create the Web application within the _layouts folder for the SharePoint site. For more information about creating this type of Web application, see How to: Modify Configuration Settings for an Application to Coexist with Windows SharePoint Services.

In a variation for this federated connector solution, you can add support for multiple external search sites by modifying the ASPX page to include details for more than one site within a case statement. The query template specified for these locations could then include a custom parameter that specifies which site in the case statement receives the federated query. Another variation is to combine the results for multiple external search providers, incorporating logic to order the results based on relevance.

Well, there're already some people who did a nice job, for example Andrew Woodward:

http://www.21apps.com/2008/01/search-server-2008-federated-sites-that.html

I would go a little further on this. Here I take Baidu as an example. Baidu is the biggest Internet search engine in China. (Google China? God knows where them are. Baidu introduced many interesting applications that Chinese users love to use. But Google China, is only famous for stealing the input method dictionary of another major Internet company SOHU, and then made its own Pinyin input method. After this was exposed to the public, they did a not so honest "apologize" and said that were two interns who did it. Perfect, later this became a popular phase in China, if anyone did evil things but was discovered by the public, he would say it's intern's or temporary employees' fault. Well, what a shame on this "not to be evil" company. - little off topic) .

Baidu.com does not return any RSS feed. What's more, it is using GB2312 encoding method to show the results. So if you directly use regex to capture something in Baidu, you will get some squares which do not make sense.

And there're some limitations in asp.net Request.QueryString method. It cannot correctly process Gb2312 encoding. So the Page Load Method must be changed to the following code:

protected void Page_Load(object sender, EventArgs e)
    {
        if (Request.QueryString["q"]!= null)
        {
            query = Request.Url.Query.ToString();
            query = query.Remove(0,3);
        }
    }

In this way, a query string will be kept so you can process it with Encode and Decode. If you use QueryString, you will get a stupid behavior that it incorrectly use Decode method in a wrong encoding charset...The result is a disater. Stupid, stupid, stupid. I want to slap the guy who wrote this method. Does he know there're not only English in this world?

For example, my nickname opal, in Chinese is 猫眼石. If queried from IE, it will be encoded using UTF-8. But Baidu can only consume GB-2312.

In UTF-8, 猫眼石 is %E7%8C%AB%E7%9C%BC%E7%9F%B3.

In GB2312, 猫眼石 is %C3%A8%D1%DB%CA%AF.

It's quite different. If you want do a search for %E7%8C%AB%E7%9C%BC%E7%9F%B3, and it is treaten as a GB2312 string, it will become 4.5 Chinese charactors. and none of them will make sense.

Okay, compain less, do more. So then we need to decode query string.

 private string getRssItemXml(string query)
    {
        //first you must decode it as UTF8. Because when IE access a utf-8 based website, it will pass the corresponding encoded strings.
        //Of course, you can modify web.config to make this application using Gb2312, but that doesn't make sense.
        query = HttpUtility.UrlDecode(query, Encoding.UTF8);
        //Then we need do encode it to gb2312. Baidu can only consume that.
        query = HttpUtility.UrlEncode(query, Encoding.GetEncoding("gb2312"));
        string url = string.Format("http://www.baidu.com/s?wd={0}", query);

        WebClient client = new WebClient();
        byte[] byteData = client.DownloadData(url);
        //Returned results are also in GB2312, so you have to rebuild it.
        string strData = Encoding.GetEncoding("gb2312").GetString(byteData);
        Regex searchPattern = new Regex("\\)\" href=\"(?<link>.*?)\" target=\"_blank\"><font size=\"3\">(?<title>.*?)</font></a><br>(?<desc>.*?)<br>");
        StringBuilder sb = new StringBuilder();

        foreach (Match m in searchPattern.Matches(strData))
        {
            sb.AppendFormat("<item><title><![CDATA[{0}]]></title><link><![CDATA[{1}]]></link><description><![CDATA[{2}]]></description></item>",m.Groups["title"].Value,m.Groups["link"].Value, m.Groups["desc"].Value);
        }

        return sb.ToString();
    }

So then put this aspx file to a website, have your federated search webpart point to it, like http://www.abc.com/Baidu.aspx?q={searchTerms}, and then you can get Baidu federated search in Microsoft Search Server 2008.

I put part of my work here:

http://cid-8007edf5c56fc334.skydrive.live.com/self.aspx/Microsoft%20Search%20Server/CaptureWeb.rar

It contains:

Baidu Federated Search Web Service
Baidu News Federated Search Web Service
iCiba (English-Chinese Dictionary) Federated Search Web Service
Dictionary.com Federated Search Web Service

Yes! You can put dictionaries on your federated search web page so if anybody want to search a word, he will get the meaning immediately! You can also have some triggers to make this happen only with numbers or charactors, etc.

Okay...Here comes Smart Search bug fix and installation guide...

Jie Li — Tue, 26 Feb 2008 19:53:37 GMT

It's my fault to ignore it for so long a time. It is not written by me originally, but by a colleague of mine, Gang Chen. But he asked me to do a favor to create a project on codeplex, so now it is maintained by me.

http://www.codeplex.com/smartsearch

And finally, I fixed the foolish bug and replaced the Chinese charactors to English words. I spent a whole afternoon to get it installed on my WSS+Search Server Express box. Everybody can try it here.

http://www.mssearch.cn:5000/Search/results.aspx?k=lotus

A installation guide is also there. I think a experienced sharepoint user can install it within 10~30 minutes.

It is quite interesting that the installation guide is also the 11111th release on Codeplex. :)

The reason I spent so much time is, WSS is using Windows Internal Database. In this code, we need to create a seperate sql table in content db, so it failed of course. To workaround this problem, you need to manually modify the code and point it to a Sql Server(Express) instance.

Try Microsoft Search Server Express for Federated Search

Jie Li — Mon, 25 Feb 2008 09:05:22 GMT

I just installed a Search Server Express on my public box, so everyone can try it just by several clicks! If you don't have the time to install one for yourself, now it's your chance to get your hands on it.

Play with default interface:

http://www.mssearch.cn:5000/Search/default.aspx

Example:

http://www.mssearch.cn:5000/Search/results.aspx?k=lotus

Play with Yahoo Image, Youtube and Flickr tag search:

http://www.mssearch.cn:5000/Search/Sandbox/media.aspx

Example:

http://www.mssearch.cn:5000/Search/Sandbox/media.aspx?k=%E5%A4%A9%E5%9D%9B

Will add more funny stuff later...

And don't forget to check here for more information.

http://www.microsoft.com/enterprisesearch

Next steps of SharePoint Search Enhancement, and...

Jie Li — Sun, 17 Feb 2008 10:53:00 GMT

I was collecting feedbacks from many people these days about the enhancement we did for SharePoint Search in China. Good feedback, bad feedback, that's okay. I just put the whole list here:

Search As You Type, SAYT: The live search feel for your SharePoint search box.

TODO: Check if there's any chance we can make use of AJAX.NET.

http://www.codeplex.com/SearchAsYouType

Predefined Search: Some kind of Saved Search. Just by a single click, and your query will be remembered and can be share to the public.

TODO: Currently none. If you have ideas, contribute to the open source project!

http://www.codeplex.com/MOSSPredefinedSearch

Smart Search: 1. Display top ten hot search keyword. 2. Display relevant search keywords.

TODO: Bugfix, and a pure English version.

http://www.codeplex.com/smartsearch

SharePoint Search Admin: A GUI based tool which can do SharePoint search administration, much better and easy to use than SharePoint Search settings page itself.

TODO: Bugfix, add more tricks so that can help service people to deliver things on time.

http://www.codeplex.com/searchadmin

Chinese "did you mean" webpart: Deliver a did you mean feature for Simplified Chinese.

TODO: None

http://www.codeplex.com/cndidyoumean

Some people have a concern that these codeplex tools are not supported by Microsoft. That's true. But I must emphasize here, that codeplex thing is OPEN SOURCE. Yes, we are talking about Open Source in Microsoft. Personally I am some kind of open source lover, I think if you really want to do some thing like SOA or SaaS, open source development model would be a pretty good alternative to a complex API set. The pain of API set is the blackbox development. Product team people may not understand the business needs as deeply as we field people do. So, it is quite possible that they do something unreasonable with the product. The gain of API set is also the blackbox model. It simplified the process of development, lowered the entry barrier of newbie developers. Of course, open source may lead to bad documentation and unreadable code, also complexity of the code you write to extend it. So, a balance between opensource and api would be better for service development. So far, Firefox has been proven to be a good example.

A little off topic. But things really are different when you talk to IT pro. It depends on what is the model: a product only solution or a product+service solution? For platforms like SharePoint, a product+service solution is definitely the wiser choice. Because in a more successful implementation, nearly everything needed to be customized for the organization. In this way, the codeplex thing would be a great help.

It may be hard to change people's thought immediately, but time would prove this.

Dealing with custom content source in SharePoint Search

Jie Li — Sat, 09 Feb 2008 20:04:25 GMT

By default, SharePoint 2007 family (MOSS/MSS) supports the following content sources: Web sites, SharePoint Sites (WSS/SharePoint Server), Exchange Public Folders(through OWA), File Shares, Lotus Notes databases on Domino Server. Through BDC and user profiles/My sites, Microsoft Office SharePoint Server 2007 has two more content sources: databases and people profiles.

Yes, we can use Protocol Handlers(It is also called "connectors", don't be confused, protocol handler is not a friendly name to most of the people so we changed it) to index other contents. A protocol handler can be implemented in C++, and also C# if you don't care about the performance. So, you can give the database ability to Search Server 2008, you can connect some other things like FileNet and Documentum... By creating a protocol handler, you can also have the ability to control security trimming, a much better and wiser way than custom security trimmer.

But how can I add/remove/edit a custom content source after the new protocol handler is registered?

The answer is: using object model. You can find CustomContentSource under Microsoft.Office.Server.Search.Administration namespace. I have a open source administration tool at http://www.codeplex.com/searchadmin (SharePoint Search Admin).

There's an interesting member in CustomContentSource: Tag. By modifying this property, you can set an URL of the page to modify the settings for a custom content source. Don't forget to append CustomContentSource.Id after the page, otherwise how can the page know which content source should it open?

For example, you can specify http://moss:90/Edit.aspx as a edit page. If the Id property of current content source is 45, the actrual link would be something like http://moss:90/Edit.aspx?cid=45.

Last but not the least important, custom content source is not limited to only new protocol handlers. You can also add something like file://, http:// or even notes:// to it.

So why not make a content source edit page for Lotus Notes? Yes! You can replace the original lame one and use your own! Check out SharePoint Search Admin, you will find the option is already there for you:)

(Still jet-lagging...See you in Seattle!)

SharePoint Search Admin v.06: Support 250K Lotus Notes Databases Now

Jie Li — Sat, 02 Feb 2008 19:57:40 GMT

http://www.codeplex.com/searchadmin

We all know that SharePoint 2007 and Microsoft Search Server 2008 support Lotus Notes content source. So we can index the databases on Domino Servers.

Y-E-S. But there ARE some issue you may face in the real world.

The original idea of SharePoint Search Admin (or MOSS Search Admin, MSA) comes from a customer request in Australia. They wanted to deal with Domino Document Manager, and they had thousands of Notes DBs to index. But in SharePoint Search Settings page, you need to manually add all these Notes dbs one by one. So I wrote MSA to batch add all these DBs.

But later, I found MSA can solve many problems when you work on enterprise search solutions built on SharePoint, for example some troubleshooting work, provide some workaround to bugs... It's more and more useful, especially in large Notes envoriment.

We will talk about the problems later, now we will only discuss this update.

Yesterday a issue happened in my customer envoriment. After they created 500 content sources for 500 Notes DBs, they cannot create any new content sources any more.

Okay, there's a 500 limit in content source per SSP. This is okay for file share and websites, because we can have 500 start address per content source. But things are very tough in Notes content source, you can only have ONE Notes DB per content source if you are using default content source creation page!

This made me crazy. We always have thousands of DBs to index, but now we can only index 500 of them. Customer don't want to have multiple SSPs.

Thanks to our great Mitch Prince, he immediately shed the light on me: Using OM.

In fact, SharePoint Lotus Notes content source can support multiple DB/Directory in single content source. The only reason we cannot do it OOB is: that Content source creation page are really poorly made. This can be workaround by using object model, exactly how MSA did.

Damn, how can I forget my own tool?

So I added this in MSA. It's quite simple, just a single line of code.

LotusNotesCS.StartAddresses.Add(startAddress);

So now you can add up to 250,000 Lotus Notes DBs in one SSP! Well, if you are sure you won't hit our performance barrier.

What about the future? In next major release of SharePoint Search Admin, I'll complete reorganize the UI, and add more useful functions to make it a powerful alternative to SharePoint Search Setting UI. Hope I can did better than System Center!

Faceted Search 2.0 for SharePoint/Search Server

Jie Li — Wed, 23 Jan 2008 09:27:12 GMT

Leonid Lyublinski released his second major version of Faceted Search today, this is a great tool that can instantly improve your SharePoint Server/Search Server user experience!

It provide the following benefits:

Grouping search results by facet

Displaying a total number of hits per facet value

Refining search results by facet value

Update of the facet menu based on refined search criteria

Displaying of the search criteria in a Bread Crumbs

Ability to exclude the chosen facet from the search criteria

Flexibility of the Faceted search configuration and its consistency with MOSS administration

and in version 2, he enhanced:

Multi- thread processing. 1st thread runs for up to 500 facets synchronously, while the 2nd thread is running asynchronously against up to ~30,000 facets

Client side refresh (not AJAX based) that updates only Facets web part w/o page refresh

Web part connection to pass Facet Settings to the Bread Crumbs

Extended facet schema now supports:

Facet icons. Default icon per Facet name complimented by an icon per Facet value
Friendly names for facet values
Built-in wild-card match, e.g. FileExtension=”ASPX?ID%” will match all “ASPX?ID=” records (useful for exclusions)
Exclusions. Allow exclude facet when values match pattern
Improved search syntax, added supports for sentences and quoted phrases

I would suggest that everyone who use SharePoint and Search Server to install this open source and free tools, it's really great.

http://www.codeplex.com/FacetedSearch

Live demo(lower right corner)

http://www.wssdemo.com/Search/Pages/results.aspx?k=search