Paresh's blog

Issue in crawl after certificate is updated in FAST Search for SharePoint 2010

Paresh Sen — Tue, 31 Jul 2012 17:54:00 GMT

Recently we observed that our crawl was continuing for a long time without any progress. On looking at the eventlog we found the below error:

"Failed to connect to SERVER Failed to initialize session with document engine: Unable to resolve Contentdistributor."

We found that FAST certificate had expired. We updated the certificate and crawl ran without issue. however, next day crawl started behaving as earlier. this time the error logged in eventlog was different:

"Failed to connect to SERVER. Failed to initialize session with document engine: Unable to find master Contentdistributor. Ensure that all Contentdistributors in the ESP installation are listed when creating icontent_manager."

On investigating we found that during the updation of the certificate, new certificate was generated on the admin node and non-admin node was still retaining the old certificate.

Issue:

Certificate was not properly updated in all the FAST non-admin nodes.

Resolution :

Exported the certificate from admin node
imported into non-admin node where secondary content distributor was running
stopped fast services on non-admin node
ran installer\scripts\replacedefaultcertificate.ps1 -thumbprint "certthumbprint"
started fast services on non-admin node

More details on managing certificates can be found here.

After full crawl no data is being retrieved: issues with index in FAST Search for SharePoint 2010

Paresh Sen — Mon, 23 Apr 2012 10:25:41 GMT

We recently came across this issue, we wanted to clean up our old indexes and build our environment. We rest the index from SharePoint and FAST server and did a full crawl. We observed that after the crawl is complete there were no data being retrieved in our searches. On looking further we observed that the FASTSearch conetnet collection and crawl logs are indicating that all the data has been crawled and indexed.

We performed an incremental crawl and amazingly we got some of the data searchable i.e. whatever data was re-crawled was not retrievable. Looking at the indexer info we got something similar to below details:

Here XXXXX = YYYYY + ZZZZZ.

This clearly showed that there were some issue with the indexes. We needed to re-index the data to get the issue fixed. we used the below command for the same:

indexeradmin resetindex -a

Once the processing completed, we got all the data back in search.

Using SharePoint 2010 Search webservice in Javascript for FAST search for sharepoint 2010

Paresh Sen — Sat, 31 Mar 2012 18:52:00 GMT

Here we will see a simple example of calling SharePoint 2010 search web service in javascript. This example is for FAST search for SharePoint 2010, so we will use FQL in our query packet.

We will be using XMLHttpRequest object for our request and response.

We need to undertand the below XML structures for moving ahead:

1] SOAP request for search service. Here, the request XML is for 'QueryEX' method. 'queryXml' will contain the actual query packet for search, as indicated by string.

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Body>
    <QueryEx xmlns="http://microsoft.com/webservices/OfficeServer/QueryService">
      <queryXml>string</queryXml>
    </QueryEx>
</soap:Body>
</soap:Envelope>

Let's assume this is represented by variable 'requestXML'.

2] SOAP response for search service. Here, the response XML is for 'QueryEX' method. 'QueryExResult' will contain the result, as indicated by xml.

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Body>
    <QueryExResponse xmlns="http://microsoft.com/webservices/OfficeServer/QueryService">
      <QueryExResult><xsd:schema>schema</xsd:schema>xml</QueryExResult>
    </QueryExResponse>
</soap:Body>
</soap:Envelope>

Let's assume this is represented by variable 'responseXML'.

3] FQL Query Packet for search, indicating the search criteria, refiners, sorting, etc.

<QueryPacket Revision=\"1000\">
<Query>
    <Context>
      <QueryText language=\"en-US\"type=\"FQL\">string(\"-\",Mode=\"AND\")</QueryText>
    </Context>
    <SupportedFormats Format=\"urn:Microsoft.Search.Response.Document.Document\"/>
    <ResultProvider>FASTSearch</ResultProvider>
    <Range>
      <StartAt>1</StartAt>
      <Count>20</Count>
    </Range>
    <EnableStemming>true</EnableStemming>
    <EnableSpellCheck>Suggest</EnableSpellCheck>
    <IncludeSpecialTermsResults>true</IncludeSpecialTermsResults>
    <IncludeRelevantResults>true</IncludeRelevantResults>
    <ImplicitAndBehavior>true</ImplicitAndBehavior>
    <TrimDuplicates>true</TrimDuplicates>
    <Properties>
      <Property name=\"Rank\"/>
      <Property name=\"Title\" />
      <Property name=\"Author\"/>
      <Property name=\"Size\" />
      <Property name=\"Path\" />
      <Property name=\"Write\"/>
      <Property name=\"SiteName\" />
      <Property name=\"HitHighlightedSummary\"/>
      <Property name=\"HitHighlightedProperties\"/>
      <Property name=\"ContentClass\"/>
      <Property name=\"IsDocument\"/>
      <Property name=\"Url\" />
      <Property name=\"FileExtension\"/>
      <Property name=\"SpSiteUrl\"/>
      <Property name=\"docvector\"/>
      <Property name=\"fcocount\" />
      <Property name=\"fcoid\"/>
    </Properties>
</Query>
</QueryPacket>

Let's assume this is represented by variable 'queryPacket'. this will replace the value string in step [1] for requestXML.

We can write a function incorporating all the above for calling the service and getting the response as follows:

function RetrieveSearchData()

{

//get the request object

var requestObject = new XMLHttpRequest();

//Initializes request and specifies the method, URL, and other information for the request.

requestObject.open("POST", "http://SERVER:PORT/_vti_bin/search.asmx", false);

//set request header

requestObject.setRequestHeader("Content-Type", "text/xml; charset=utf-8");

//Get FQL queryPacket along with the SOAP envelope

var packet = requestXML;

//Sends an HTTP request to the server and receives a response

requestObject.send(packet);

//process responseXML

}

There are multiple ways to achieve the above processing in AJAX, ASP.net or traditional C#. This example is just one of the ways to process SharePoint webservice in javascript code.

Case study on Search driven applications for PLM and DAM systems

Paresh Sen — Tue, 20 Mar 2012 05:16:00 GMT

My post on search driven applications can be found here.

Developing Windows 8 app: Accessing configuration files deployed as part of deployment

Paresh Sen — Fri, 03 Feb 2012 20:57:00 GMT

While developing your windows 8 app, you may need default configurations. In order to make things more generic, you may choose to keep your configuration in a separate xml to be deployed along with the app. Here, we will see how we can access a file deployed as part of deployment and use it.

In windows 8, access to folder system is provided by api's that are async in nature and 'Task' based. Therefore, we will use 'async' methods for this operation.

private async Task<SampleData> ReadFromConfigurationFile()

{

try

{

//get the package reference

Windows.ApplicationModel.Package package = Windows.ApplicationModel.Package.Current;

//get installed location detail

Windows.Storage.StorageFolder storageFolder = package.InstalledLocation;

// Use serializer class

XmlSerializer serializer = new XmlSerializer(typeof(SampleData));

try

{

//file variable

StorageFile sampleFile = null;

try

{

//Get the file

sampleFile = await storageFolder.GetFileAsync("ConfigurationStored.xml");

}

catch (FileNotFoundException ex)

{

//handle exception

}

//open file in read mode

var file = await sampleFile.OpenAsync(FileAccessMode.Read);

//Read stream

Stream inStream = Task.Run(() => file.OpenRead()).Result;

return (SampleData)serializer.Deserialize(inStream);

}

catch (Exception ex)

{

//handle exception

}

catch (Exception ex)

{

//handle exception

}

return null;

}

Note the use of await and async pattern for handling the asynchronous operation.

If you want to modify the default value and write the file back, open file in write mode and after modification save the same at the installed location.

More sample on file access can be found at http://code.msdn.microsoft.com/windowsapps/File-access-sample-d723e597.

Implementing BCS connector based on enumerator methods

Paresh Sen — Wed, 14 Dec 2011 19:47:00 GMT

Business Connectivity Services (BCS) in SharePoint 2010 is all about accessing data external to SharePoint. BCS provides read/write access to external data from line-of-business (LOB) systems, Web services, databases, and other external systems within Microsoft SharePoint 2010 and Microsoft Office 2010 applications. BCS enhances the SharePoint platform’s capabilities with out-of-box features, services and tools that streamline development of solutions with deep integration of external data and services. Details can be found here.

The core function of BDC is to provide connectivity support to the following types of external systems:

Databases
Web/WCF services
Microsoft .NET Framework connectivity assemblies
Custom data sources

There are scenarios when we wish to do custom implementation to achieve the desired processing. This can be achieved by developing Visual Studio based code solutions. We add up code to do the required processing for implementing List\Item operations. These solutions work great and provide a lot of flexibility.

However, in scenarios where BCS is used to pull data and index in SharePoint for implementing search driven applications, we need constant updation of data from the external system. In such systems we need incremental update to work to ensure CRUD operations. In such a scenario we need to create solutions that implement ChangedIDEnumerator\DeletedIDEnumerator apart from general List (IDEnumerator), Item (ReadItem) operations.

IDEnumerator - Used for list operation

ChangedIDEnumerator - Identifies the modified items

DeletedIDEnumerator - Identifies the deleted items

ReadItem - used for Item operation

apart from the above we may need to implement streamaccessor method for ingesting streams for indexing like documents.

Here, we will have a look on the methods and the mapping in model file.

IDEnumerator

Syntax: public static IEnumerable<EntityType> IDEnumerator()

Returns an enumerator of EntityType. This will have all the items that need to be indexed. This can be infered as the finder method.

Model file mapping:

            <Method IsStatic="false" Name="IDEnumerator">
              <Parameters>
                <Parameter Direction="Return" Name="returnParameter">
                  <TypeDescriptor TypeName="System.Collections.Generic.IEnumerable`1[EntityType Class details]" IsCollection="true" Name="ListName">
                    <TypeDescriptors>
                      ...
                    </TypeDescriptors>
                  </TypeDescriptor>
                </Parameter>
              </Parameters>
              <MethodInstances>
                <MethodInstance Type="IdEnumerator" ReturnParameterName="returnParameter" Default="true" Name="EnumeratorName" DefaultDisplayName="DisplayName">
                  <Properties>
                    <Property Name="RootFinder" Type="System.String">x</Property>                    
                  </Properties>
                </MethodInstance>
              </MethodInstances>
            </Method>

ChangedIDEnumerator

Syntax: public static IEnumerable<EntityType> ChangedIDEnumerator(DateTime LastRunDate)

Returns an enumerator of EntityType. it contains the modified items. LastRunDate indicates time that will be used as reference for this crawl for finding the modified items. We can map this value or can use another parameter (configuration parameter, or reading from external ways) for getting the reference.

Model file mapping:

         <Method IsStatic="false" Name="ChangedIDEnumerator">
              <FilterDescriptors>
                <FilterDescriptor Name="LastRunDate" Type="InputOutput">
                  <Properties>
                    <Property Name="SynchronizationCookie" Type="System.String">LastModifiedDate</Property>
                  </Properties>
                </FilterDescriptor>
                <FilterDescriptor Name="FilterDescriptor" Type="Timestamp" />
              </FilterDescriptors>
              <Parameters>
                <Parameter Name="@LastRunDate" Direction="InOut">
                  <TypeDescriptor Name="LastRunDateTypeDescriptor" TypeName="System.DateTime" AssociatedFilter="LastRunDate">
                    <Interpretation>
                      <NormalizeDateTime LobDateTimeMode="Local" />
                    </Interpretation>
                  </TypeDescriptor>
                </Parameter>
                <Parameter Direction="Return" Name="returnParameter">
                  <TypeDescriptor TypeName="System.Collections.Generic.IEnumerable`1[EntityType Class details]" IsCollection="true" Name="ListName">
                    ...
                  </TypeDescriptor>
                </Parameter>
              </Parameters>
              <MethodInstances>
                <MethodInstance Type="ChangedIdEnumerator" ReturnParameterName="returnParameter" Default="true" Name="OperationName" DefaultDisplayName="DisplayName" />
              </MethodInstances>
            </Method>

DeletedIDEnumerator

Syntax: public static IEnumerable<EntityType> DeletedIDEnumerator(DateTime LastRunDate)

Returns an enumerator of EntityType. it contains the deleted items. LastRunDate indicates time that will be used as reference for this crawl for finding the deleted items. We can map this value or can use another parameter (configuration parameter, or reading from external ways) for getting the reference.

        <Method IsStatic="false" Name="DeletedIDEnumerator">
              <FilterDescriptors>
                <FilterDescriptor Name="LastRunDate" Type="InputOutput">
                  <Properties>
                    <Property Name="SynchronizationCookie" Type="System.String">LastModifiedDate</Property>
                  </Properties>
                </FilterDescriptor>
                <FilterDescriptor Name="FilterDescriptor" Type="Timestamp" />
              </FilterDescriptors>
              <Parameters>
                <Parameter Name="@LastRunDate" Direction="InOut">
                  <TypeDescriptor Name="LastRunDateTypeDescriptor" TypeName="System.DateTime" AssociatedFilter="LastRunDate">
                    <Interpretation>
                      <NormalizeDateTime LobDateTimeMode="Local" />
                    </Interpretation>
                  </TypeDescriptor>
                </Parameter>
                <Parameter Direction="Return" Name="returnParameter">
                  <TypeDescriptor TypeName="System.Collections.Generic.IEnumerable`1[EntityType class details]" IsCollection="true" Name="ListName">
                    <TypeDescriptors>
                      ...
                    </TypeDescriptors>
                  </TypeDescriptor>
                </Parameter>
              </Parameters>
              <MethodInstances>
                <MethodInstance Type="DeletedIdEnumerator" ReturnParameterName="returnParameter" Default="true" Name="OperationName" DefaultDisplayName="DisplayName" />
              </MethodInstances>
            </Method>

ReadItem

Syntax: public static Entity ReadItem(string ID)

Returns type entity. it contains the detail of an item. ID is the unique identifier for the item.

Mapping in model:

       <Method IsStatic="false" Name="ReadItem">
              <Parameters>
                <Parameter Name="ID" Direction="In">
                  <TypeDescriptor Name="ID" TypeName="System.String" IdentifierEntityName="ENTITY" IdentifierEntityNamespace="NAMESPACE" IdentifierName="ID">
                  </TypeDescriptor>
                </Parameter>
                <Parameter Name="returnParameter" Direction="Return">
                  ...
                </Parameter>
              </Parameters>
              <MethodInstances>
                <MethodInstance Name="ReadItem" Type="SpecificFinder" ReturnParameterName="returnParameter" Default="true" DefaultDisplayName="DisplayName">
                  <Properties>
                    <Property Name="LastDesignedOfficeItemType" Type="System.String">None</Property>
                    <Property Name="WindowsSecurityDescriptorField" Type="System.String">SecurityDescriptor</Property>
                  </Properties>
                </MethodInstance>
              </MethodInstances>
            </Method>

Here, SecurityDescriptor is used for implementing security at record level.

StreamAccessor

Syntax: public Stream GetStream(string ID)

GetStream can be used if we intend to ingest documents in the crawl process. ID is the unique identifier.

Mapping in Model:

          <Method IsStatic="false" Name="GetStream">
              <Parameters>
                <Parameter Name="ID" Direction="In">
                  <TypeDescriptor Name="ID" TypeName="System.String" IdentifierEntityName="ENTITY" IdentifierEntityNamespace="NAMESPACE" IdentifierName="ID" />
                </Parameter>
                <Parameter Name="returnParameter" Direction="Return">
                  <TypeDescriptor Name="returnParameterTypeDescriptor" TypeName="System.IO.Stream" />
                </Parameter>
              </Parameters>
              <MethodInstances>
                <MethodInstance Name="MethodName" Type="StreamAccessor" ReturnParameterName="returnParameter">
                  <Properties>
                   ...
                  </Properties>
                </MethodInstance>
              </MethodInstances>
            </Method>

More on custom connectors can be found here.

Another great post on BCS can be found here.

Set-ExecutionPolicy : Windows PowerShell updated your execution policy successfully, but the setting is overridden by a policy defined at a more specific scope!!!

Paresh Sen — Wed, 07 Dec 2011 14:50:00 GMT

You have reached to this blog post probably after hitting into the below issue on executing:

Set-ExecutionPolicy "RemoteSigned"

Error thrown may be as follows:

Set-ExecutionPolicy : Windows PowerShell updated your execution policy successfully, but the setting is overridden by a policy defined at a more specific scope. Due to the override, your shell will retain its current effective execution policy of "Unrestricted". Type "Get-ExecutionPolicy -List" to view your execution policy settings. For more information, please see "Get-Help Set-ExecutionPolicy.

"Set-ExecutionPolicy" cmdlet is used for changing the user preference for the Windows PowerShell execution policy. details can be found here.

If the current logged-in user is an administrator on the system with proper access but still getting the above issue, following blogpost instruction may be helpful in re-setting group policy.

However, if the issue persists and "Get-ExecutionPolicy -List" command shows something similar as shown below:

    MachinePolicy = RemoteSigned
    UserPolicy      = Undefined
    CurrentUser     = Unrestricted
    LocalMachine = RemoteSigned

Execute the below commands in powershell commandline as an administrator:

Set-ExecutionPolicy "RemoteSigned" -Scope Process -Confirm:$false

Set-ExecutionPolicy "RemoteSigned" -Scope CurrentUser -Confirm:$false

This will set the execution policy for the given scope.

On executing "Get-ExecutionPolicy -List", we should now see the below:

    MachinePolicy = RemoteSigned
    UserPolicy       = Undefined
    Process           = RemoteSigned
    CurrentUser     = RemoteSigned
    LocalMachine = RemoteSigned

How to increase relevancy of FAST search results for "Phrase" query

Paresh Sen — Sat, 30 Apr 2011 15:20:05 GMT

Fast Search for SharePoint 2010 indexes data fed to the system during crawl. on query these indexes are used to retrieve relevant results. However, when the indexed data is very high and "Phrase" search is used we may not get as relevant results as expected.

For example, in indexed data few words like "ABC" are very common, phrase searches involving these words may show less relevant results than expected. Suppose we search with ABC XYZ or "ABC XYZ" or "XYZ ABC", ideally we should get different result set based on the existence of these phrases in indexed data. However, "Phrase" search behavior changes for very frequent indexed keywords like ABC and it treats the Phrase "ABC XYZ" as ABC AND XYZ.

Expected behavior can be ensured by changing PositionStopWordThreshold and StopWordThreshold values for RankProfile in FAST admin node. New value should be changed based on the specific scenario of the target farm and volume of data. Default value for Rank profile are listed below:

Rank Profile parameter	Original value
PositionStopWordThreshold	20000000
StopWordThreshold	2000000

Follow this msdnlink for details.

In order to do the changes, need to execute below in FAST Server management powershell:

$RankProfile = Get-FASTSearchMetadataRankProfile -Name <RankProfileName>

$RankProfile.StopWordThreshold = <new integer value>

$RankProfile.PositionStopWordThreshold = <new integer value>

$RankProfile.Update()

Restart FAST services.

There are few downside to these changes and must be analyzed before the changes( Follow this msdnlink for details)

Increase in CPU usage
Slower Search Performance

Crawl fails with error "The item may be too large or corrupt. You may also verify that you have the latest version of this IFilter"

Paresh Sen — Wed, 06 Apr 2011 09:10:44 GMT

Our crawl failed for large files with crawl error : "The item may be too large or corrupt. You may also verify that you have the latest version of this IFilter".

In eventviewer of the crawl component observed error: "The search service stopped the filter daemon because it was consuming too many resources. A new daemon will automatically be started, and no user action is required."

This got resolved following the suggested changes for increasing registry values for DedicatedFilterProcessMemoryQuota, FilterProcessMemoryQuota, FolderHighPriority.

Resolving error "No snap-ins have been registered for windows powershell version 2"

Paresh Sen — Wed, 06 Apr 2011 08:18:53 GMT

After doing some update\configuration changes in SharePoint Farm, I was getting this issue when trying to open "SharePoint 2010 Management Shell":

Add-PSSnapin : No snap-ins have been registered for windows powershell version 2

This was resolved by changing the path of the powershell.exe used to initiate the management shell. In properties -> Shortcut -> Target, powershell.exe was pointing to sysWOW64 folder. I changed the same to point to System32 [c:\windows\system32\windowspowershell\v1.0\powershell.exe].

Module (WebAnalyzer) at “server[FQDN]:13300” is not responding

Paresh Sen — Fri, 25 Mar 2011 18:15:00 GMT

We came across an interesting issue related to crawl in SharePoint\FAST farm. In one of the farm set-up with distributed nodes for both SharePoint and FAST, crawl was running very slow. It was taking almost 5 times the actual time of completion. All services in SharePoint and FAST were running as expected and system resources were well within acceptable range.

In eventviewer of FAST admin node we observed consistent entry for the warning: Module (WebAnalyzer) at “server[FQDN]:13300” is not responding.

Looking at the webanalyzer.log file under var\log\webanalyzer we found below entries:

Couldn't connect to the ConfigServer
Missing nctrls on these hosts

"nctrl status" was showing webanalyzer as running, however "waadmin showstatus" failed to retrieve the details.

This issue turn out to be due to mismatch in hostname while configuring the multiple nodes in FAST farm. FAST installation is case sensitive wrt to server names. This msdn reference talks about the same.

We fixed this issue following the below steps:

stop services "nctrl stop"
Edit etc\waconfig.xml and etc\Node.xml to ensure that hostnames are in the same case
start services "nctrl start"

This is done in all the nodes of the farm.

Case-insensitive sort order in FAST Search for SharePoint 2010

Paresh Sen — Tue, 08 Mar 2011 13:35:00 GMT

By default FAST search sort is based on ASCII sort order. Therefore, sorting will follow [0-9A-Za-z] as pattern i.e. Numeric values will be followed by capital letter followed by small letter words in ascending sort.

In order to change the order to be case insensitive, we need to a change the fullsort profile. Index-profile contains details for each managed metadata property and sort profile needs to be changed in the same place. In the FAST installation folder inside index-profiles locate the file "deployment-ready-index-profile.xml". [ We should take back up of existing file before any changes, so that we can revert back to original settings as needed]

Add this for the relevant property: <fullsort profile="generic" />

For example:

deafult value: <field fullsort="yes" ..................... name="PropertyName" description=""/>

Modified value: <field fullsort="yes" ..................... name="PropertyName" description=""><fullsort profile="generic" /></field>

After doing a full crawl data will be presented in case-insensitive order for the given property. We can further set the order as ascending or descending as required.

This post has a similar discussion on this subject.

One of the limitation of this implementation is that once a new metadata managed property is added or an existing one gets deleted the index-profile file is re-generated. The newly generated file does not retain the modfied values.

Crawl fails to index data and doesn't show any update in crawl log

Paresh Sen — Mon, 07 Mar 2011 14:14:00 GMT

I observed this issue recently for a SharePoint\Fast Search for SharePoint 2010 farm set up. Crawl was going into a infinite state without any errors and updates. There were no indication in crawl log for any success or failures. It was all working fine and there was no change to system\farm setup.

I tried verifying SharePoint\Fast services, everything seems to be working fine. Therefore, I tried accessing the default search url (http://localhost:13280) in FAST and performed a simple search. The search resulted in an error with description "All partitions are down Code: 1003". On looking further in eventlog found the error "We are low on disk space, waiting for disk space to be freed".

It turned out to be the disk space issue.The fast installation drive was having much lower disk space than the recommended one.

How to enable security trimming on search result for external data indexed in SharePoint 2010: SQL connector

Paresh Sen — Tue, 22 Feb 2011 12:55:00 GMT

In continuation to my last post I'll explain how to enable security trimming on search results in SharePoint 2010 for external database. We need to have a BDC model file created with suitable operations for a table that is being indexed in SharePoint. More details on creation of BDC model file can be found here:

Create an External Content Type Based on a SQL Server Table [link]
Business Data Connectivity Models [link]

At this point we assume that we have the below items in place:

External data table with all relevant columns that need to be indexed
BDC model file with List\item operations defined in it for the above table

In order to have item level security implemented and search result trimming based on security, we need to have the security descriptor generated for each record and present in the table for each row. If the external system is already not having this data available, we can generate the same using:

private Byte[] GetSecurityDescriptor(string domain, string username) {
    NTAccount acc = new NTAccount(domain, username);    SecurityIdentifier sid = (SecurityIdentifier)acc.Translate(typeof(SecurityIdentifier));
   CommonSecurityDescriptor sd = new CommonSecurityDescriptor(false, false, ControlFlags.None,sid, null, null, null);
   sd.SetDiscretionaryAclProtection(true, false);

    //Deny access to all users.
   SecurityIdentifier everyone = new SecurityIdentifier(WellKnownSidType.WorldSid, null);
   sd.DiscretionaryAcl.RemoveAccess(AccessControlType.Allow, everyone, unchecked((int)0xffffffffL), InheritanceFlags.None, PropagationFlags.None);

    //Grant full access to a specified user.
   sd.DiscretionaryAcl.AddAccess(AccessControlType.Allow, sid, unchecked((int)0xffffffffL), InheritanceFlags.None, PropagationFlags.None);
    byte[] secDes = new Byte[sd.BinaryLength];
   sd.GetBinaryForm(secDes, 0);

    return secDes;
}

 
 Follow this msdn link for details.

This can be used for a user or group or can be modified as required for a set of users\groups that can be mapped to windows users\groups. This will generate a byte stream and we need to store it in the table for each row as a varbinary value.

Once we set the table data appropriately, we need to map this new column in our BDC model file. Assume the column containing security details is called "SecurityDescriptor". we will use this name here forward for mapping.

In the BDC model file do the below changes:

In "Read List" section, for Finder method ensure the properties as shown:

In "Read Item" section, ensure we have a parameter for "SecurityDescriptor" column:

In "Read Item" section, for "SpecificFinder" method, ensure mapping for "SecurityDescriptor" with property "WindowsSecurityDescriptorField"

                  <MethodInstance Type="SpecificFinder" ReturnParameterName="TableRead Item" ReturnTypeDescriptorPath="TableRead Item[0]" Default="true" Name="TableRead Item" DefaultDisplayName="Read Item Table">
                    <Properties>
                         <Property Name="LastDesignedOfficeItemType" Type="System.String">None</Property>
                         <Property Name="WindowsSecurityDescriptorField" Type="System.String">SecurityDescriptor</Property>
                    </Properties>
                  </MethodInstance>

After importing the BDC model file, run a full crawl on the content source. Now search results should show the security trimmed data.

Crawl of external data in SharePoint 2010 is going into hung state?

Paresh Sen — Tue, 22 Feb 2011 11:19:00 GMT

Recently we have encountered this issue where our crawl process was going into hung state. We have a SharePoint 2010\FAST farm and we have used BCS custom connector to crawl external data with large no of documents.We have used the metadata extraction for the documents. Our crawl process was going into hung state and was not completing even after many hours.

Issues observed during crawling of data were:

connection time out
data time out
failure in indexing large files
mssdmn process becomes unresponsive
high memory\CPU utilization

After a long troubleshooting we were able to isolate a couple of issues that were adding to this behavior.

Large files [Files greater that 64 MB in size]
Low system disk space ["Temp" folder is used for temporary download of files for indexing in case of custom connector where download functionality is written in code]

Steps to rectify the crawl behavior in above scenario:

Increase the ConnectTimeout, DataTimeout values

Increase "MaxDownloadSize" to some more appropriate value as per the data

Follow these links[ blog post , blog post ] for guidance on how to change the above values.

Enabling security trimming on search results: External data in SharePoint 2010

Paresh Sen — Fri, 18 Feb 2011 18:52:00 GMT

Business Connectivity Services (BCS) in SharePoint 2010 provides the capability to connect to external data and create scalable application utilizing this out-of-box feature.

The core function of BDC is to provide connectivity support to the following types of external systems:

Databases
Web/WCF services
Microsoft .NET Framework connectivity assemblies
Custom data sources

Data present in the external repository may have access restrictions i.e. based on role\group\user data can have restricted access. This means every user may not have access to every record present in the repository. SharePoint 2010 provides security trimming that ensures that information is available to the appropriate users only. For each record access information must be present and should be mapped to the users\groups identifiable in SharePoint.

In order to enable security trimming on the search results, do the below:

Generate security descriptor for access at record level for each item present in external repository
Create a BDC model with appropriate method instances for reading item and list details
Return the security descriptor generated as a property to the BDC model along with your item details
Map the security descriptor property with "WindowsSecurityDescriptorField"

Security descriptor is byte array generated for a user\group with appropriate access rights defined in it. This value is mapped with the "WindowsSecurityDescriptorField" in "SpecificFinder" for security to work.

In my next post, I'll walk through a sample database example for the above steps.