Jie Li's GeekWorld : SharePoint Search Admin

SharePoint Search - Lotus Notes Indexing Best Practice

Jie Li — Tue, 01 Jul 2008 08:35:40 GMT

Many people have been asking for the best practice or a guide to properly maintain Lotus Notes indexing function in SharePoint Search. So here it is, this is not a official guide, but our experience in several big customers. I will write this in a Q/A format, so you can navigate to see which question applies to your current problem.

Q1. How many Lotus Notes content source can I crawl at the same time?
A1: One content source per Domino Server. If all of your stuff are put on a single Domino Server, you have to crawl them one by one. But If you have several Domino servers to index, then you can index them at the same time. This is a limitation of IBM Lotus Notes C++ API. So you may need to carefully set schedules to crawl these content sources.

Q2. How many Lotus Notes content source shall I crawl at the same time?
A2: The only difference from the 1st question is CAN/SHALL. There should be a limit on this number, but what is this number? I don’t have the direct answer for the question, because this number depends on your hardware performance, memory usage, network legacy and bandwidth…. so many factors. For a recent hardware with 8GB ram, I would recommend 3,with scheduled memory recycling – we will talk about this later.

Q3. I have a Notes database indexed, but how come the time of full crawl is nearly the same with incremental crawl?
A3: During an incremental crawl, SharePoint search engine will check LastModifiedTime property of target documents/items, and to determine if the target object should be fully retrieved back to its index. However, for certain content source, this property is not retrieved or mapped to something else by mistake, therefore, the engine can only get all the content back to check if there’s any difference. I’m checking a possible solution for this problem, and will update if I can find something.

Q4. Should I use x86 or x64 for Lotus Notes indexing?
A4: Because of the limitation of IBM Notes C++ API, Notes Protocol Handler can only run on a x86 box. However, you can still use x64 query servers and WFEs. Remember: the same tier should not be mixed with x64/x86 boxes, but you can have x86 indexer tier with x64 query and x64 wfe tiers, this is recommended for Notes search in SharePoint 2007/Search Server 2008. (IBM released x64 version of their API recently, but it’s impossible to make current NotesPH to work with that, many things changed)

Q5. You mentioned memory recycling – what does that mean?
A5: Due to x86 limitation, the memory per process is limited to certain number. And because we are calling Notes client through API, it’s quite possible MSSEARCH/MSSDMN process will hit memory limit after a crawl of large numbers of documents. So I recommend you to recycle these processes for every certain amount of time. This can prevent possible stuck of the crawl. In order to do this, you might need to write your own schedule program with SharePoint search administration APIs, and restart osearch service when it’s need. I will also add this function to SharePoint Search Admin 0.81 and later in a few days.

Q6. Any ideas about security trimming support? What should I do in Domino side?
A6: You can use Lotus Notes users and groups to control security, and map them to AD users to achieve search result security trimming in SharePoint. But it is generally advised to not use Lotus Notes Roles for security control, as there’s no correspond thing in active directory.

Q7. To be added.
A7:

Btw, I’m moving to a new position in IW PMG, as a Technical Product Manager to drive SharePoint IT Pro readiness. So in future there would be more things like SharePoint Governance appear on this blog:).

SharePoint Search Admin 0.80b – Now with Keywords and Bestbets Backup and Restore Feature

Jie Li — Mon, 02 Jun 2008 03:17:00 GMT

It has been a long time since last update. In this version of SharePoint Search Admin, I added a feature by very popular request: Keywords and Bestbets.

I mark it a “Beta” because there’re still something I didn’t finish due to time limitation, so I disabled “Add Keyword” and “Add Bestbet” button. They are not the focus because using SSA to add them cannot provide a better experience than default SharePoint UI.

Let’s take a look at the interface.

Still ugly :) I promise I’ll improve it.

So you can remove individual Keyword, remove all keywords by a single click, backup and restore whole keywords/bestbets collection in a few seconds with this new feature.

A sample XML generated by SSA Keywords backup.

Index FTP content with SharePoint 2007/Search Server 2008

Jie Li — Thu, 10 Apr 2008 10:03:00 GMT

Question: Can SharePoint 2007/Search Server 2008 index and search ftp?

Answer: OOTB you cannot. But we have a FTP protocol handler in SharePoint 2001 ResKit, and that one can be used on SharePoint 2007 or Search Server 2008.

Important Notice: This approach is NOT supported by Microsoft. It is just for your test purpose only, and should not be used in an important production envoriment.

Some restrictions may apply: x86 and anonymous ftp only.

Here's the steps.

1. Get a copy of ftpph.dll. It is in SharePoint 2001 ResKit.

2. Copy it to c:\Program Files\Microsoft Office Servers\12.0\Bin

3. Register this dll. Open command prompt, navigate to c:\Program Files\Microsoft Office Servers\12.0\Bin, run "regsvr32 ftpph.dll".

4. Open Regedit, navigate to HKLM\SOFTWARE\Microsoft\Office Server\12.0\Search\Setup\Protocol Handlers

5. Insert a new string item: "ftp". The value is "FtpPH.SearchProtocol.1".

6. In command prompt, restart search service by "net stop osearch" and "net start osearch"

7. Download SharePoint Search Admin at http://www.codeplex.com/SearchAdmin , and then add a new custom content source for your ftp site.

8. Start full crawl, and you will see the crawl log. Job done!

Next steps of SharePoint Search Enhancement, and...

Jie Li — Sun, 17 Feb 2008 10:53:00 GMT

I was collecting feedbacks from many people these days about the enhancement we did for SharePoint Search in China. Good feedback, bad feedback, that's okay. I just put the whole list here:

Search As You Type, SAYT: The live search feel for your SharePoint search box.

TODO: Check if there's any chance we can make use of AJAX.NET.

http://www.codeplex.com/SearchAsYouType

Predefined Search: Some kind of Saved Search. Just by a single click, and your query will be remembered and can be share to the public.

TODO: Currently none. If you have ideas, contribute to the open source project!

http://www.codeplex.com/MOSSPredefinedSearch

Smart Search: 1. Display top ten hot search keyword. 2. Display relevant search keywords.

TODO: Bugfix, and a pure English version.

http://www.codeplex.com/smartsearch

SharePoint Search Admin: A GUI based tool which can do SharePoint search administration, much better and easy to use than SharePoint Search settings page itself.

TODO: Bugfix, add more tricks so that can help service people to deliver things on time.

http://www.codeplex.com/searchadmin

Chinese "did you mean" webpart: Deliver a did you mean feature for Simplified Chinese.

TODO: None

http://www.codeplex.com/cndidyoumean

Some people have a concern that these codeplex tools are not supported by Microsoft. That's true. But I must emphasize here, that codeplex thing is OPEN SOURCE. Yes, we are talking about Open Source in Microsoft. Personally I am some kind of open source lover, I think if you really want to do some thing like SOA or SaaS, open source development model would be a pretty good alternative to a complex API set. The pain of API set is the blackbox development. Product team people may not understand the business needs as deeply as we field people do. So, it is quite possible that they do something unreasonable with the product. The gain of API set is also the blackbox model. It simplified the process of development, lowered the entry barrier of newbie developers. Of course, open source may lead to bad documentation and unreadable code, also complexity of the code you write to extend it. So, a balance between opensource and api would be better for service development. So far, Firefox has been proven to be a good example.

A little off topic. But things really are different when you talk to IT pro. It depends on what is the model: a product only solution or a product+service solution? For platforms like SharePoint, a product+service solution is definitely the wiser choice. Because in a more successful implementation, nearly everything needed to be customized for the organization. In this way, the codeplex thing would be a great help.

It may be hard to change people's thought immediately, but time would prove this.

SharePoint Search Admin v.06: Support 250K Lotus Notes Databases Now

Jie Li — Sat, 02 Feb 2008 19:57:40 GMT

http://www.codeplex.com/searchadmin

We all know that SharePoint 2007 and Microsoft Search Server 2008 support Lotus Notes content source. So we can index the databases on Domino Servers.

Y-E-S. But there ARE some issue you may face in the real world.

The original idea of SharePoint Search Admin (or MOSS Search Admin, MSA) comes from a customer request in Australia. They wanted to deal with Domino Document Manager, and they had thousands of Notes DBs to index. But in SharePoint Search Settings page, you need to manually add all these Notes dbs one by one. So I wrote MSA to batch add all these DBs.

But later, I found MSA can solve many problems when you work on enterprise search solutions built on SharePoint, for example some troubleshooting work, provide some workaround to bugs... It's more and more useful, especially in large Notes envoriment.

We will talk about the problems later, now we will only discuss this update.

Yesterday a issue happened in my customer envoriment. After they created 500 content sources for 500 Notes DBs, they cannot create any new content sources any more.

Okay, there's a 500 limit in content source per SSP. This is okay for file share and websites, because we can have 500 start address per content source. But things are very tough in Notes content source, you can only have ONE Notes DB per content source if you are using default content source creation page!

This made me crazy. We always have thousands of DBs to index, but now we can only index 500 of them. Customer don't want to have multiple SSPs.

Thanks to our great Mitch Prince, he immediately shed the light on me: Using OM.

In fact, SharePoint Lotus Notes content source can support multiple DB/Directory in single content source. The only reason we cannot do it OOB is: that Content source creation page are really poorly made. This can be workaround by using object model, exactly how MSA did.

Damn, how can I forget my own tool?

So I added this in MSA. It's quite simple, just a single line of code.

LotusNotesCS.StartAddresses.Add(startAddress);

So now you can add up to 250,000 Lotus Notes DBs in one SSP! Well, if you are sure you won't hit our performance barrier.

What about the future? In next major release of SharePoint Search Admin, I'll complete reorganize the UI, and add more useful functions to make it a powerful alternative to SharePoint Search Setting UI. Hope I can did better than System Center!