Background & Symptoms:

• There is a SharePoint 2007 large list with over 300,000 items in it
• Items are arranged into different folder to make sure each folder contains less than 2000 items in it
• Half of the items need to be set permission to restrict only groups of users to access so they are put into restricted folder and permission is applied on the folder level
• SharePoint group is used to control the permission
• In the pilot phase, under 10 users are using it and they are testing this application
• SharePoint incremental crawl sometimes finish in 5 minutes … even we have many content changes
• And sometimes the incremental crawl will take more than several hours … even there is no content change at all

What could be the problem?

By analyzing the ULS log, we found out that when it took long time to crawl, the security only crawl is happening. Check this blog about how to detect security only crawl: http://blogs.msdn.com/b/russmax/archive/2009/02/09/troubleshooting-security-only-crawl.aspx

Why is this happening intermittently?

Because it’s in the pilot/testing phase, test users are moving themselves from one SharePoint group to another and that will trigger the security only crawl!

What is the solution?

Do not use SharePoint group and use AD group instead.  From SharePoint point of view, when you move users from one security group to another the group SID never change. If you have multiple DCs you need to be aware that the permission setting will take effect after the replication is completed when you move users among groups.