Jie Li's GeekWorld : Domino

SharePoint Search - Lotus Notes Indexing Best Practice

Jie Li — Tue, 01 Jul 2008 08:35:40 GMT

Many people have been asking for the best practice or a guide to properly maintain Lotus Notes indexing function in SharePoint Search. So here it is, this is not a official guide, but our experience in several big customers. I will write this in a Q/A format, so you can navigate to see which question applies to your current problem.

Q1. How many Lotus Notes content source can I crawl at the same time?
A1: One content source per Domino Server. If all of your stuff are put on a single Domino Server, you have to crawl them one by one. But If you have several Domino servers to index, then you can index them at the same time. This is a limitation of IBM Lotus Notes C++ API. So you may need to carefully set schedules to crawl these content sources.

Q2. How many Lotus Notes content source shall I crawl at the same time?
A2: The only difference from the 1st question is CAN/SHALL. There should be a limit on this number, but what is this number? I don’t have the direct answer for the question, because this number depends on your hardware performance, memory usage, network legacy and bandwidth…. so many factors. For a recent hardware with 8GB ram, I would recommend 3,with scheduled memory recycling – we will talk about this later.

Q3. I have a Notes database indexed, but how come the time of full crawl is nearly the same with incremental crawl?
A3: During an incremental crawl, SharePoint search engine will check LastModifiedTime property of target documents/items, and to determine if the target object should be fully retrieved back to its index. However, for certain content source, this property is not retrieved or mapped to something else by mistake, therefore, the engine can only get all the content back to check if there’s any difference. I’m checking a possible solution for this problem, and will update if I can find something.

Q4. Should I use x86 or x64 for Lotus Notes indexing?
A4: Because of the limitation of IBM Notes C++ API, Notes Protocol Handler can only run on a x86 box. However, you can still use x64 query servers and WFEs. Remember: the same tier should not be mixed with x64/x86 boxes, but you can have x86 indexer tier with x64 query and x64 wfe tiers, this is recommended for Notes search in SharePoint 2007/Search Server 2008. (IBM released x64 version of their API recently, but it’s impossible to make current NotesPH to work with that, many things changed)

Q5. You mentioned memory recycling – what does that mean?
A5: Due to x86 limitation, the memory per process is limited to certain number. And because we are calling Notes client through API, it’s quite possible MSSEARCH/MSSDMN process will hit memory limit after a crawl of large numbers of documents. So I recommend you to recycle these processes for every certain amount of time. This can prevent possible stuck of the crawl. In order to do this, you might need to write your own schedule program with SharePoint search administration APIs, and restart osearch service when it’s need. I will also add this function to SharePoint Search Admin 0.81 and later in a few days.

Q6. Any ideas about security trimming support? What should I do in Domino side?
A6: You can use Lotus Notes users and groups to control security, and map them to AD users to achieve search result security trimming in SharePoint. But it is generally advised to not use Lotus Notes Roles for security control, as there’s no correspond thing in active directory.

Q7. To be added.
A7:

Btw, I’m moving to a new position in IW PMG, as a Technical Product Manager to drive SharePoint IT Pro readiness. So in future there would be more things like SharePoint Governance appear on this blog:).

Real World Lotus Notes Index Result in SharePoint 2007

Jie Li — Wed, 20 Feb 2008 17:47:51 GMT

Crawled Notes data: 1240 Gbytes

Crawled Notes Items: 2.7 Million

SharePoint Index: 70 Gbytes

SharePoint Search DB: 120 Gbytes

Data vs Index = 124:7 = 18:1, which is a ratio of 6%.

Index vs Search DB = 7: 12. Search DB is 160% of Index.

In total, (Search DB+Index)/Data=16%

Remember, this ratio varies in different environment, depends on how you use Notes Applications, how many attachments are there, and many other factors. I put it here only as a reference. Maybe update later.

SharePoint Search Admin v.06: Support 250K Lotus Notes Databases Now

Jie Li — Sat, 02 Feb 2008 19:57:40 GMT

http://www.codeplex.com/searchadmin

We all know that SharePoint 2007 and Microsoft Search Server 2008 support Lotus Notes content source. So we can index the databases on Domino Servers.

Y-E-S. But there ARE some issue you may face in the real world.

The original idea of SharePoint Search Admin (or MOSS Search Admin, MSA) comes from a customer request in Australia. They wanted to deal with Domino Document Manager, and they had thousands of Notes DBs to index. But in SharePoint Search Settings page, you need to manually add all these Notes dbs one by one. So I wrote MSA to batch add all these DBs.

But later, I found MSA can solve many problems when you work on enterprise search solutions built on SharePoint, for example some troubleshooting work, provide some workaround to bugs... It's more and more useful, especially in large Notes envoriment.

We will talk about the problems later, now we will only discuss this update.

Yesterday a issue happened in my customer envoriment. After they created 500 content sources for 500 Notes DBs, they cannot create any new content sources any more.

Okay, there's a 500 limit in content source per SSP. This is okay for file share and websites, because we can have 500 start address per content source. But things are very tough in Notes content source, you can only have ONE Notes DB per content source if you are using default content source creation page!

This made me crazy. We always have thousands of DBs to index, but now we can only index 500 of them. Customer don't want to have multiple SSPs.

Thanks to our great Mitch Prince, he immediately shed the light on me: Using OM.

In fact, SharePoint Lotus Notes content source can support multiple DB/Directory in single content source. The only reason we cannot do it OOB is: that Content source creation page are really poorly made. This can be workaround by using object model, exactly how MSA did.

Damn, how can I forget my own tool?

So I added this in MSA. It's quite simple, just a single line of code.

LotusNotesCS.StartAddresses.Add(startAddress);

So now you can add up to 250,000 Lotus Notes DBs in one SSP! Well, if you are sure you won't hit our performance barrier.

What about the future? In next major release of SharePoint Search Admin, I'll complete reorganize the UI, and add more useful functions to make it a powerful alternative to SharePoint Search Setting UI. Hope I can did better than System Center!