Background & Symptoms:
What could be the problem?
By analyzing the ULS log, we found out that when it took long time to crawl, the security only crawl is happening. Check this blog about how to detect security only crawl: http://blogs.msdn.com/b/russmax/archive/2009/02/09/troubleshooting-security-only-crawl.aspx
Why is this happening intermittently?
Because it’s in the pilot/testing phase, test users are moving themselves from one SharePoint group to another and that will trigger the security only crawl!
What is the solution?
Do not use SharePoint group and use AD group instead. From SharePoint point of view, when you move users from one security group to another the group SID never change. If you have multiple DCs you need to be aware that the permission setting will take effect after the replication is completed when you move users among groups.