Create a custom XML indexing connector for SharePoint 2013
Having a way to feed XML content from a network share or local file directory into the search index is highly useful. This blog post describes how to install and deploy a custom XML connector to the Search service application. The custom XML connector will enable you to crawl content from a network share or local directory, and feed it to the search index. When content is in the search index, it can be displayed in a public facing web site through search.
The follow-up blog post “Display XML content in a publishing site when using the custom XML indexing connector for SharePoint 2013” will describe how you can configure a publishing site to display your XML content through search.
The diagram below shows how a custom XML connector feeds XML content from a network share and local file directory to the search index. This content is then shown in Search Web Parts on a publishing site.
Required components
Before you begin, make sure that you have installed the following:
- SharePoint 2013 Server
- Visual studio 2012
Install and deploy a custom XML indexing connector for SharePoint Server 2013
In the procedure below, you will extract the files needed to create the custom XML extractor, and install this on your local SharePoint Server 2013 installation on the default Search service application.
Note: The user that performs this operation has to be an administrator for the Search service application.
To install the custom XML connector for SharePoint Server 2013
- Download the .zip file at the bottom of this blog. The file includes the following:
-A Microsoft Visual Studio Project to install and deploy the custom XML connector.
-A PowerShell script to create a term set on a publishing site.
-A text file that contains the taxonomy of the term set.
-An .xml file that contains sample .xml product data.
- Extract the contents to a folder on your computer, for example c:\CustomXMLConnector .
- On the Start menu, click All Programs, click Microsoft Visual Studio 2010, and then click Visual Studio Tools and open a Visual Studio command prompt.
- To install the XmlFileConnector.dll, at the command prompt, type the following command and run it:
gacutil /i <ExtractedFolderPath>\bin\Debug\XmlFileConnector.dll
where:
<ExtractedFolderPath> is the folder where your extracted files are located, for example c:\CustomXMLConnector.
Expected result after running this command is "Assembly successfully added to the cache".
- To add the correct key and values to the registry, double click the file named xmldoc.reg
- On the Start menu, click All Programs, then click Microsoft SharePoint 2013 Products and open a SharePoint 2013 Management Shell as an administrator.
- To configure the custom XML connector, at the command prompt, type the following command and run it:
- $searchapp = Get-SPEnterpriseSearchServiceApplication -Identity "Search Service Application"
- New-SPEnterpriseSearchCrawlCustomConnector -SearchApplication $searchapp –Protocol xmldoc -Name xmldoc –ModelFilePath "<ExtractedFolderPath>\Model.xml" where:
<ExtractedFolderPath> is the folder where your extracted files are located, for example c:\CustomXMLConnector. The Model.xml file is included in the Visual Studio project.
- To confirm the configuration, at the command prompt, type the following command and run it:
Get-SPEnterpriseSearchCrawlCustomConnector -SearchApplication $searchapp
Expected result:
- To restart the search service, at the command prompt, type the following commands and run them:
net stop osearch15
net start osearch15
Create new crawled property category for the custom XML connector
When the custom XML connector crawls content, the crawled properties discovered during crawl will have to be added to a crawled property category. You will have to create this category.
Note: The user that performs this operation has to be an administrator for the Search service application.
To create a new crawled property category for the custom XML connector
- On the Start menu, click All Programs, then click Microsoft SharePoint 2013 Products and open a SharePoint 2013 Management Shell as an administrator.
- To create a new crawled property category, at the command prompt type the following commands and run them, where: <ConnectorName> is the name you want to give the custom XML connector, for example Custom XML Connector:
- $searchapp = Get-SPEnterpriseSearchServiceApplication -Identity "Search Service Application"
- New-SPEnterpriseSearchMetadataCategory -Name "<ConnectorName>" -Propset "BCC9619B-BFBD-4BD6-8E51-466F9241A27A" -searchApplication $searchapp
The Propset GUID, BCC9619B-BFBD-4BD6-8E51-466F9241A27A, is hardcoded in the file XmlDocumentNamingContainer.cs and should not be changed.
- To specify that if there are unknown properties in the newly created crawled property category, these should be discovered during crawl, at the command prompt, type and run the following:
$c = Get-SPEnterpriseSearchMetadataCategory -SearchApplication $searchapp -Identity "<ConnectorName>"
$c.DiscoverNewProperties = $true
$c.Update()
where:
<ConnectorName> is the name you want to give the custom XML connector, for example Custom XML Connector.
- To confirm the creation of the crawled property category, on the home page of the SharePoint Central Administration website, in the Application Management section, click Manage service applications, and then Search service application. On the Search Service Application page, in the Queries and Results section, click Search Schema and then Categories.
Create new content source for the XML content
To specify what, when and how the XML content should be crawled, you have to create a new content source for your XML content.
Note: The user that performs this operation has to be an administrator for the Search service application.
To create a new content source for the XML content
- On the home page of the SharePoint Central Administration website, in the Application Management section, click Manage service applications
- On the Manage Service Applications page, click Search service application.
- On the Search Service Administration Page, in the Crawling section, click Content Sources.
- On the Manage Content Sources page, click New Content Source.
- On the Add Content Source page, in the Name section, in the Name box, type a name for the new content source, for example XML Connector.
- In the Content Source Type section, select Custom repository.
- In the Type of Repository section, select xmldoc.
- In the Start Address section, in the Type start addresses below (one per line) box, type the address from where the crawler should being crawling the XML content. The start address syntax is different depending on where the XML content is located:
- If the XML content is located on a local drive, use the following syntax:
xmldoc://localhost/<XMLcontentfolder>/#x=doc:id;;urielm=url;;titleelm=title# where:
<XMLcontentfolder> is the location of your XML content, for example C$/XMLCATALOG.
You can also change the variables doc:id,url and title. To use our example file, you should type: xmldoc://localhost/C$/XMLCATALOG/#x=Product:ID;;titleelm=Title;;urlelm=Url#
- If the XML content is located on a local drive, use the following syntax:
-
If the XML content is located on a network drive, use the following syntax:
xmldoc://<SharedNetworkPath>/#x=doc:id;;urielm=url;;titleelm=title#
where:
<SharedNetworkPath> is the location of your XML content, for example C$/XMLCATALOG.
You can also change the variables doc:id,url and title. To use our example file, you should type: xmldoc://localhost/C$/XMLCATALOG/#x=Product:ID;;titleelm=Title;;urlelm=Url#
Important: Make sure the crawler account has access to the local drive or network share.
Note: In the specified address, xmldoc is a protocol that was added to the registry. The part enclosed in #, contains properties that will be extracted from the document and will be associated with document ID, title and navigation url. #x=Product:ID matches the content in the .xml sample file that you will use later.
- Verify that your newly created content source is shown on the Search Service Application page.
Create sample XML content
Now that you have successfully installed and deployed the custom XML connector, you can use it to extract XML content and display this through search on a publishing site.
To create XML sample content
- Create a folder on your computer, for example C:/xmlcatalog.
- Create an .xml sample file with following content, and store it in your newly created folder.
<?xml version="1.0" encoding="utf-8" ?>
<Products>
<Product>
<ID>C234</ID>
<Title>Contoso series2</Title>
<ItemCategoryText>Cameras</ItemCategoryText>
<Url>https://<server_name>/sites/pub1/Pages/DetailsPage.aspx?id=C234</Url>
<Category>Cameras/SLR</Category>
<Price>799</Price>
<Description>This camera is making great pictures. Easy to use to an affordable price. </Description>
</Product>
<Product>
<ID>C235</ID>
<Title>Contoso Camera</Title>
<ItemCategoryText>Cameras</ItemCategoryText>
<Url>https://<server_name>/sites/pub1/Pages/DetailsPage.aspx?id=C235</Url>
<Category>Cameras/compact</Category>
<Price>350</Price>
<Description>This camera is making great pictures. Easy to use to an affordable price. </Description>
</Product>
</Products>
To enable crawled items from XML document to appear in CBS with managed navigation make sure to include a field ( e.g. ItemCategoryText.) which will be mapped to the taxonomy term set.
Each record should include a field which will point to the associated category. An item called “Contoso series2” should point to “Cameras”.Then it will be possible to associate menu selection and results in CBS.
The publishing site, on which you want to display this content, uses managed navigation (I will show you how to import a term set to drive managed navigation later in this blog). By specifying a query in a Content Search Web Part that contains terms from your managed navigation term set, you can display indexed items that are associated with each term. To do this, a field in the .xml file has to have the same value as a term in the term set that drives your managed navigation. In our sample .xml file, the field ItemCategoryText contains this value. This value specifies to which term in the term set the entry belongs. For example, for the item with the title “Contoso series2”, the value for ItemCategoryText is “Cameras”.So, when users navigate to the "Cameras" page on the publishing site, the Content Search Web Part can display all entries that have the value "Cameras" for ItemCategoryText (I will show you how to specify the query in the Content Search Web Part later in this blog).
To crawl newly created content source
Now that you have some sample .xml content, you will need to crawl this content.
- On the home page of the SharePoint Central Administration website, in the Application Management section, click Manage service applications
- On the Manage Service Applications page, click Search Service Application.
- On the Search Service Administration Page, in the Crawling section, click Content Sources.
- Start a full crawl of the content source you created earlier.
- When crawling is complete, ensure that crawled properties where created. On the Search Service Application page, in the section Queries and Results, click Search Schema, and then Categories. Your newly created category should contain 17 crawled properties.
To see the crawled properties, click on the category name.
- Verify that a crawled property was created for the ItemCategoryText field by typing ItemCategoryText in the Crawled properties box, and then click thesearch arrow.
To create a new managed property for ItemCategoryText
Now that you have crawled properties for the .xml content, you have to create a managed property for the field ItemCategoryText. You will use this managed property when you define the query in a Content Search Web Part.
- On the Search Service Application page, click Managed Properties.
- On the Managed Properties page, click New Managed Property.
- In the Property name field, type a name for the managed property, for example ItemCategoryText.
- Check the following check boxes:
- Searchable
- Queryable
- Retrievable
- Allow multiple values
- In the Mappings to crawled properties section, click the button Add a Mapping.
- In the Search for a crawled property name section, type ItemCategoryText, and click Find.
- In the Select a crawled property section, click ItemCategoryText, and OK.
Verify the mapping by searching for the managed property ItemCategoryText, and see that it is mapped to ItemCategoryText property.
In order to add the new managed property to the search index, you have to do an incremental of the content source.
To start an incremental crawl
- On the home page of the SharePoint Central Administration website, in the Application Management section, click Manage service applications
- On the Manage Service Applications page, click Search Service Application.
- On the Search Service Administration Page, in the Crawling section, click Content Sources.
- Start an incremental crawl of the content source you created earlier.(see figure below)
- When the incremental crawl has complete, verify that the crawl completed without any errors. In the Diagnostics section, click Crawl Log, and verify that no errors were found.
Using a bigger product set.
The .zip file also includes a bigger product set, containing 800 items.
As you now have a custom XML connector that works:
- Copy the Product.xml file to your content source location (C:\XMLCATALOG\)
- Initiate a new incremental crawl as stated in the previous section.
- When the incremental crawl has complete, verify that the crawl completed without any errors. In the Diagnostics section, click Crawl Log, and verify that no errors were found.
You have now crawled the xml document from your local drive, created and mapped the proper managed property (ItemCategorytext) to the field Itemcategirytext in the xml file, and indexed it.
This blog post ends here.
If you are interested in seeing how you can set up a publishing site to display your XML content through search, please visit our blog post: “Display XML content in a publishing site when using the custom XML indexing connector for SharePoint 2013”.
Acknowledgements
Authors: Anders Fagerhaug (andersfa) and Armen Kirakosyan (armenk)
Please also see the blog post from Nadeem Ishqair on the same subject for SharePoint 2010.