Jie Li's GeekWorld : Security Trimming

SharePoint Search - Lotus Notes Indexing Best Practice

Jie Li — Tue, 01 Jul 2008 08:35:40 GMT

Many people have been asking for the best practice or a guide to properly maintain Lotus Notes indexing function in SharePoint Search. So here it is, this is not a official guide, but our experience in several big customers. I will write this in a Q/A format, so you can navigate to see which question applies to your current problem.

Q1. How many Lotus Notes content source can I crawl at the same time?
A1: One content source per Domino Server. If all of your stuff are put on a single Domino Server, you have to crawl them one by one. But If you have several Domino servers to index, then you can index them at the same time. This is a limitation of IBM Lotus Notes C++ API. So you may need to carefully set schedules to crawl these content sources.

Q2. How many Lotus Notes content source shall I crawl at the same time?
A2: The only difference from the 1st question is CAN/SHALL. There should be a limit on this number, but what is this number? I don’t have the direct answer for the question, because this number depends on your hardware performance, memory usage, network legacy and bandwidth…. so many factors. For a recent hardware with 8GB ram, I would recommend 3,with scheduled memory recycling – we will talk about this later.

Q3. I have a Notes database indexed, but how come the time of full crawl is nearly the same with incremental crawl?
A3: During an incremental crawl, SharePoint search engine will check LastModifiedTime property of target documents/items, and to determine if the target object should be fully retrieved back to its index. However, for certain content source, this property is not retrieved or mapped to something else by mistake, therefore, the engine can only get all the content back to check if there’s any difference. I’m checking a possible solution for this problem, and will update if I can find something.

Q4. Should I use x86 or x64 for Lotus Notes indexing?
A4: Because of the limitation of IBM Notes C++ API, Notes Protocol Handler can only run on a x86 box. However, you can still use x64 query servers and WFEs. Remember: the same tier should not be mixed with x64/x86 boxes, but you can have x86 indexer tier with x64 query and x64 wfe tiers, this is recommended for Notes search in SharePoint 2007/Search Server 2008. (IBM released x64 version of their API recently, but it’s impossible to make current NotesPH to work with that, many things changed)

Q5. You mentioned memory recycling – what does that mean?
A5: Due to x86 limitation, the memory per process is limited to certain number. And because we are calling Notes client through API, it’s quite possible MSSEARCH/MSSDMN process will hit memory limit after a crawl of large numbers of documents. So I recommend you to recycle these processes for every certain amount of time. This can prevent possible stuck of the crawl. In order to do this, you might need to write your own schedule program with SharePoint search administration APIs, and restart osearch service when it’s need. I will also add this function to SharePoint Search Admin 0.81 and later in a few days.

Q6. Any ideas about security trimming support? What should I do in Domino side?
A6: You can use Lotus Notes users and groups to control security, and map them to AD users to achieve search result security trimming in SharePoint. But it is generally advised to not use Lotus Notes Roles for security control, as there’s no correspond thing in active directory.

Q7. To be added.
A7:

Btw, I’m moving to a new position in IW PMG, as a Technical Product Manager to drive SharePoint IT Pro readiness. So in future there would be more things like SharePoint Governance appear on this blog:).

When, why and how to deal with Custom Security Trimmer in Enterprise Search? - Part II

Jie Li — Wed, 07 Nov 2007 17:23:00 GMT

In part II we will go through the code a little deeper.

Check permission against different systems

Please open this page,

http://msdn2.microsoft.com/en-us/library/aa981173.aspx

Look at this part of the code.

for (int x = 0; x < crawlURLs.Count; x++)
{
/*
To fully implement the security trimmer, add code to perform the security check and determine if strUser can access crawlURLs[x]. 
If strUser can access crawlURL[x], then:
*/
retArray[x] = true;
//If not:
retArray[x] = false;
}

Quite simple explanation. But how can you ?

1. Use WindowsIdentity.GetCurrent().Name to get current username, or if you are using FBA, that is HttpContext.Current.User.Identity.Name.

2. Then use this username to check with the target system, if he has the permission to crawlURL[x], then return a True.

Different system has different security checking method. Here' re some suggested ways to check security:

Content source	Method
Web Sites, with SQL Server in backend	Directly use System.Data.Sqlclient to deal with the database and get the permission
Web Sites, with Oracle in backend	System.Data.OracleClient. You must install Oracle Client first.
Web Sites, with DB2 in backend	DB2 .Net Data Provider ODBC
Web Sites, with MySQL in backend	MySQLDriverCS http://sourceforge.net/projects/mysqldrivercs/ or MySQL connector/NET http://www.mysql.com/products/connector/net/
File Share	File.GetAccessControl SharePoint already has security trimming function built-in for file shares. It would be very uncommon that you need to deal with CST in this scenario. But be aware, if you want extra security trimming with file shares, the built-in security trimmer(the one applied in query time we talked in part I) will applied first. There's no way to get it replaced. And if you are using FBA, which means your current identity is changed from windows user to a httpcontext user, you will get nothing in your search result if the file share is not a public one.
Lotus Notes	Lotus Domino Objects, a COM object to be used in other languages

If you want to have a better performance when a CST is applied...

I suggest that you cache the permission settings to your own box and check it in CST. Remote calls may have huge impact on the performance, especially Lotus Notes. Meanwhile, check security with remote machine also means an impact to the target system. If that system is critical, this will affect customer's business.

The cache thing can be done with some small tools, of course you can write a small application by using Lotus Domino Objects and grab all the notes ACL back to a SQL table, that depends on your own opinion.

Another important thing is to set a CheckLimit in your CST. If CheckLimit is reached, CST will report something back to user, or do something you defined, and stop the check. This message can be something like "too many results pls refine your keywords", "Please try keyword1+keyword2+keyword3"....That will help.

Register a custom security trimmer

The trimmer must be compiled with strong name. You must first install it to the assembly by the following command(There're some errors in SDK):

C:\Program Files\Microsoft Visual Studio 8\SDK\v2.0\Bin>gacutil.exe /i c:\Trimmer\CustomSecurityTrimmerSample.dll /f

C:\Trimmer\CustomSecurityTrimmerSample.dll is my trimmer's path, change it with your own one.

A very important step: Create an "include" crawl rule with the URL you want to bind this CST with. If you don't create it, you cannot deploy the trimmer. In this sample, the path is http://localhost:8100/*.

Then you should deploy it with stsadm:

C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\BIN>stsadm -o registersecuritytrimmer -ssp SharedServices1 -id 2 -typeName "CustomSecurityTrimmerSample.CustomSecurityTrimmer, CustomSecurityTrimmerSample, Version=1.0.0.0, Culture=neutral, PublicKeyToken=b6c7fa67516b1230" -rulepath http://localhost:8100/*

PublicKeyToken is the token you can see in windows\assembly directory. rulepath is the crawl rule path you just created.

And don't forget iisrset. Then, if any search result matches the crawl rule, CST will be launched to check the permission.

When, why and how to deal with Custom Security Trimmer in Enterprise Search? - Part I

Jie Li — Mon, 05 Nov 2007 11:37:08 GMT

First of all, security trimming is very important in Enterprise Search. Users who have no rights to the documents should not see descriptions in their search results. They should not be aware of those items at all.

When building Enterprise Search solutions using Microsoft SharePoint Server 2007(MOSS), you can find that MOSS support file share, sharepoint, lotus notes security trimming out of the box. This means, the protocol handler picks up ACL in index time, and security trimming will be applied at query time. Such query behavior is like a SQL sentence:

SELECT * from scope() where freetext("keyword") and YourCurrentUserRight="True" (This is not the real sql sentence, just to give you an idea)

So query performance will not be impacted.

But, what about other stuff like website, database, or a custom content source?

Custom Security Trimmer(CST) is used in MOSS, to support security trimming of such things. The behavior of CST is quite different from build-in security trimmer. It is run at query time, but because the YourCurrentUserRight value is not there, CST will access target system to retrieve this value after the search results come out. It will check one by one, for example, there're 4,000 items in the result about "jokes", but you can only access 100 items. So the process changed to:

1. Do a search for "jokes", this is like SELECT * from scope() where freetext("keyword"), and no security trimming applied. 4,000 items returned in the result. This is not displayed to the user.

2. Because CST is registered with "crawl rules"(this is one of the worst name examples I had ever seen in Microsoft, wth, a rule applied at query time is called CRAWL RULE?), if the path of the item meet the rule, CST will be launched to check if current user has the permission to read this item. If he has, CST will report a "True". Note, multiple CST instances will be launched at the same time to check different items, and it seems you cannot control this number. I think it's around 4-5.

3. After the "True" number of items in one page is meet, for example 10 items CST reported True after checked against about 200 items, the first page of result will be displayed.

Let's do some basic calculation job. What will happen if a bad CST is applied? The key point is how much time will be used to check the permission in CST. If a CST will need one second to check one item, meanwhile 4 CSTs are launched, 200 items will need you 50 seconds to complete the job. This means, you have to wait for 50 seconds to get the search result showed in your browser!

Terrible, right? Even worse, if you have 100,000 items in a result array, and you only have permission against 4 items, the search service will crush because of the timeout.

So that's why CheckLimit is also needed in the implementation of CST.

Now, the best practice when you want to create a CST:

1. Reduce the time needed to check the permission. You can do some trick to make it faster, for example, store the permission mapping in a local SQL table first, and use CST to check local table not the remote one, so you can bypass the network delay.

2. Correctly set CheckLimit. Return a more user friendly message when the limit is met.

To implement a CST, you can refer to SDK, or these articles:

http://msdn2.microsoft.com/en-us/library/aa981236.aspx
http://msdn2.microsoft.com/en-us/library/aa981563.aspx

These are not very good articles, some of the information are misleading. So far as I know, there're some much better articles on the way, but I don't know the exact date of when they will be published on MSDN. Later in another post I will go through the code to explain more.