On The Search

Mashup Your Enterprise Search Results in SharePoint 2010 Using Refiners (and Bing Maps)

arnts — Tue, 09 Mar 2010 07:57:00 GMT

An enterprise search solution can provide you with fast and secure access to data that traditionally has been stored in different silos. This can e.g. be leveraged to create enterprise mashups, something that is particularly interesting together with information extraction techniques you'll find in more advanced search platforms like FAST Search. FAST Search for SharePoint 2010 has e.g. the ability to automatically recognize words and phrases by type (entities) within unstructured text, and for this blog post we'll look more at how we can leverage the locations extractor to create a simple mashup in your search results page. The locations extractor will tag each document with extracted locations (during document processing), and we'll use the Bing Maps AJAX API to geo code these locations before we plot them on a map on the search results page. Note that the locations are available in a locations refiner, and will reflect your entire search results when using FAST Search (as a "deep" refiner, see e.g. here for more on refiners - aka faceted search).

Background

To mashup your location refiner with a Bing Map we can e.g. implement a new Web Part. The approach we are taking here is very similar to what was shown in order to create a Tag Cloud and a Bar Chart Web Part, i.e. plug in a new Web Part that will work with and leverage existing search Web Parts in a search center. Use the RefinementManager to fetch the refinement results for the locations refiner, and use the Bing Maps API to geocode and plot the locations on a map. Note that the Bing Maps AJAX API is based on client side JavaScript, so we need some plumbing to load the JavaScript in the Web Part (as Web Parts are running serverside).

The screen shot below is from a FAST Search results page with a Bing Maps Mashup Web Part added.

Setup

The locations refiner is available out-of-the-box from FAST Search. It is however not configured to be fetched and displayed in the Refinement Panel Web Part by default, so we need to specify this first;

In your FAST Search Center, click Site Actions > Edit Page
Edit your Refinement Panel Web Part;

Update the Filter Category Definition XML, and add an entry to include the locations managed property

<Category Title="Location" Description="Locations found in document" Type="Microsoft.Office.Server.Search.WebControls.ManagedPropertyFilterGenerator" MetadataThreshold="1" NumberOfFiltersToDisplay="4" MaxNumberOfFilters="20" ShowMoreLink="True" MappedProperty="locations" MoreLinkText="show more" LessLinkText="show fewer" ShowCounts="Count" />

Uncheck Use Default Configuration, and Click Apply
Click Save to go back to your search page

See also previous blog entry for instructions on how to setup your Visual Studio environment.

Implementation

I've attaced some sample code to this blog post (see link below), i.e. an implementation of a Web Part called BingMashupWebPart (note that code is for illustration purposes, not meant to be ready for any production system). In the BingMashupWebPart.cs class, the method CreateChildControls() outputs a div tag for the JavaScript to be able to render the map. The OnPreRender() method contains most of the logic; it fetches any refinement results using the RefinementManager and an XPath expression, then formats the refinement values as JavaScript string. The dynamically generated JavaScript (based on refinement values) is registered on the ClientScriptManager on the Page, and so is the BingMashupScripts.js file. Quick note: to refer the JavaScript file from the Web Part you can set the Build Action property to Embedded Resource for the file.

The JavaScript file contains the functions needed to geocode each location, plot the locations on a map together with an information box, scale the map to fit all locations, and place the map on the div we added in the Web Part (see e.g. Getting a Precise Location in Bing Maps for more details on geo coding and the Bing API). Note that the Refinement Manager is shared on a page, and if we cannot assume that the Refinement Panel Web Part will be available, we must set which refiners to fetch ourselves in code. Alternatively we can extend the RefinementWebPart class. Note also that the locations refiner is configured to return the 20 most important locations for your search results in the XML above (see setup section).

Test

See previous post on how to add the Web Part to your search page.

Summary

With FAST Search for SharePoint 2010 it is easy to hook into your search results and create new and interestring mashups. This blog post has shown how to mashup values from the locations refiner with Bing Maps, but the options are many. Other "deep" refiners are e.g. companies and person names, in addition to of course the search results them selves.

Visualize Your Enterprise Search Results (and Navigation) in SharePoint 2010

arnts — Wed, 03 Mar 2010 09:14:00 GMT

A visual representation can be helpful for quickly understanding your search result. We've previously seen how to extend the user experience with a Tag Cloud Web Part, so now let's look at how to leveraging the out-of-the-box refiners to visualize the search results in a SharePoint 2010 search center. We'll walkthrough how to programmatically add a Bar Chart Web Part, a Web Part that will allow us to visualize one dimension of your search results.

Background

Refiners (aka. faceted search) was, as described on the Enterprise Search Blog, previously only available in SharePoint search through 3rd party add-ons. It is now natively supported with SharePoint 2010, and FAST Search adds the ability of "deep" refinement. This is faceted search accross result sets of any size, with precise counts on the refinement values/facets. This is typically critical in certain applications, like e.g. in research and analysis applications where the counts on facets are used for decision making.

In the image below you can see a Bar Chart Web Part added to the left zone in a FAST Search Center. This chart is based on the "format" refiner, and the rest of this blog post will show how to implement such a Web Part.

Setup

See my previous post on how to setup your development environment with Visual Studio.

Implementation

Refinement values can be accessed from the RefinementManager, a class in the Federation OM. To render a nice chart you can use the Chart control, a SharePoint wrapper around the ASP.NET Chart control. With these two components we should be well on our way.

The code below implements a Web Part called BarChartRefinementWebPart (note that the code is for illustration purposes only, not meant to be production ready). The first thing the Web Part does is set up a chart control in the CreateChildControls() method. This is so we later can add refinement values to it, and render the control. The refinement values we want to chart are fetched in the OnPreRender() method by leveraging the RefinementManager; first an XPath expression is used to select only the refinement values from the "format" refiner (i.e. an out-of-the-box managed property which has a refiner), then the refinement values are added to the chart. Finally, in the Render() method the chart is rendered (unless no refinement values are fetched).

// This code released under the terms of the

// Microsoft Public License (MS-PL, http://opensource.org/licenses/ms-pl.html.)

using System;

using System.Drawing;

using System.Web.UI;

using System.Web.UI.DataVisualization.Charting;

using System.Web.UI.WebControls.WebParts;

using System.Xml;

using Microsoft.Office.Server.Search.WebControls;

namespace MyVisualWebPartProject.BarChartRefinementlWebPart

{

public class BarChartRefinementlWebPart : WebPart

{

// the chart for visualizing

private Microsoft.Office.Server.WebControls.Chart chart;

// the managed property with a refiner to visualize

private const string refiner = "format";

protected override void CreateChildControls()

{

// setup the chart

ChartArea area = new ChartArea("My Chart Area");

// ..and make it look nice

area.AxisX.MajorGrid.LineColor = Color.LightGray;

area.AxisY.MajorGrid.LineColor = Color.LightGray;

chart = new Microsoft.Office.Server.WebControls.Chart();

chart.Width = 200;

chart.Height = 200;

chart.ChartAreas.Add(area);

chart.EnableViewState = true;

Controls.Add(chart);

}

protected override void OnPreRender(EventArgs e)

{

RefinementManager refinementManager = RefinementManager.GetInstance(this.Page);

// get the refinement values

XmlDocument refinementXmlDoc = refinementManager.GetRefinementXml();

if (refinementXmlDoc != null)

{

// collect refinement values to chart

Series refinementSeries = new Series();

refinementSeries.ChartType = SeriesChartType.Bar;

refinementSeries.Color = Color.LightSkyBlue;

refinementSeries.BorderColor = Color.LightGray;

XmlNodeList xmlFilterNodes = refinementXmlDoc.DocumentElement.SelectNodes("/FilterPanel/FilterCategory[@ManagedProperty='" + refiner + "']/Filters/Filter[Count>0]");

foreach (XmlNode filter in xmlFilterNodes)

{

string x = filter.SelectSingleNode("Value").InnerText;

string y = filter.SelectSingleNode("Count").InnerText;

int i = refinementSeries.Points.AddXY(x, y);

// display count in chart

refinementSeries.Points[i].Label = y;

// ..and make chart clickable

refinementSeries.Points[i].Url = filter.SelectSingleNode("Url").InnerText;

}

if (refinementSeries.Points.Count > 0)

chart.Series.Add(refinementSeries);

}

protected override void Render(HtmlTextWriter writer)

{

if (chart.Series.Count == 0)

{

writer.Write("<p><center>No data to render for property " + refiner + "</center></p>");

}

else

{

base.Render(writer);

}

We are dependent on refinement values being available for this Web Part to work. In the out-of-the-box search center in SharePoint, the set of refiners to fetch is set in the Refinement Panel Web Part. So as long as this Web Part is also part of your page or search center, we can leverage this. Note that the RefinementManager is shared on a page, similar to the SharedQueryManager we used for the Tag Cloud. If we cannot assume that the RefinementWebPart will be available, we can set which refiners to fetch ourselves. Alternatively we can extend the RefinementWebPart class.

Summary

This blog post has shown how to leverage an out-of-the-box refiner in SharePoint 2010 to visualize your search results. The RefinementManager in the Federation OM provides a place to hook in so you easily can create new Web Parts that will work nicely with other search Web Parts. It is the same interface for both FAST Search and SharePoint Search, but with FAST Search you get "deep" refinements, i.e. faceted search across the entire result set.

Walkthrough: Create a simple Tag Cloud Web Part based on search results

arnts — Wed, 10 Feb 2010 10:18:00 GMT

Let's see how we can extend the SharePoint 2010 search user experience by programmatically implementing a Tag Cloud Web Part. We'll use the Federation OM, the object model (OM) Web Parts in SharePoint 2010 are based on, to create a Web Part like in the screen shot below.

Background

The communication model between the search Web Parts changed from SharePoint 2007 to SharePoint 2010. Now the Web Parts use the Federation OM together with a public class called the SharedQueryManager that is shared by all synchronous Web Parts. There is one shared instance of the SharedQueryManager per search page, and through this class you can access the other classes that are part of the Federation OM (see also previous post for an overview of the query integration points). Using the Federation OM you can hook into the query path; you can e.g. fetch the search results (as in the sample code below) after the query has been executed, or you can modify the query before submitting it to the search backend (e.g. add query terms before request is submitted).

Setup and Design

We want the Tag Cloud (aka. Word Cloud) Web Part to visually display the most important terms in our result set. To keep it simple we'll leverage the document vectors of the first few results. These document vectors are created during document processing (before indexing), and are available in the managed property "docvector" when querying against FAST Search for SharePoint. A document vector indicates the most important terms/concepts in a document and the corresponding weight, i.e. like this: "[string1,weight1][string2,weight2]...[stringN,weightN]". Note also that these vectors are used for similarity search, a feature available with FAST Search on the object models.

Anyway, the goal of this walkthrough is to create a simple web part that will work nicely with the other out-of-box search web parts in SP2010. Here' are the setup steps.

To implement this new web part, I'm using Visual Studio 2010 RC1;

Open Visual Studio 2010

Download here first if needed

Create a new Visual Studio project and solution

Select e.g. the Visual Web Part or the Web Part template under SharePoint 2010 in VS2010, and click OK
Provide a URL to your FAST Search Center site, and click Finish.

Add a reference to the Microsoft.Office.Server.Search.dll in your newly created project

E.g. from C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\ISAPI

You are ready to start coding!

Implementation

I've included the code below that implements a TagCloudWebPart class. Here I access the SharedQueryManager in the OnInit() method, and keep a reference to the QueryManager. This is so we in the OnPreRender() method can access the result set, and collect the document vectors we want (assumption is that we will have one FAST Search Location available on this page). Here we also remove the surrounding brackets from these vectors, and store the terms and associated weight in a dictionary. The final part happens in the RenderContents() method where we output the terms from the dictionary, sorted descending by weight (giving a larger fontsize to terms with highest weight).

Here's the code (note: code is for illustration purposes only, not meant to be production ready)

// This code released under the terms of the

// Microsoft Public License (MS-PL, http://opensource.org/licenses/ms-pl.html.)

using System;

using System.Collections.Generic;

using System.ComponentModel;

using System.Linq;

using System.Web.UI;

using System.Web.UI.WebControls.WebParts;

using System.Xml.XPath;

using Microsoft.Office.Server.Search.Query;

using Microsoft.Office.Server.Search.WebControls;

namespace MyVisualWebPartProject.TagCloudWebPart

{

[ToolboxItemAttribute(false)]

public class TagCloudWebPart : WebPart

{

// Visual Studio might automatically update this path when you change the Visual Web Part project item.

private const string _ascxPath = @"~/_CONTROLTEMPLATES/MyVisualWebPartProject/TagCloudWebPart/TagCloudWebPartUserControl.ascx";

// To store the terms and weights used to render the tag cloud

private Dictionary<string, double> dict = new Dictionary<string, double>();

// To access the search results

private QueryManager queryManager;

protected override void OnInit(EventArgs e)

{

queryManager = SharedQueryManager.GetInstance(this.Page).QueryManager;

base.OnInit(e);

}

protected override void CreateChildControls()

{

Control control = Page.LoadControl(_ascxPath);

Controls.Add(control);

}

protected override void OnPreRender(EventArgs e)

{

LocationList locList = queryManager[0];

if (locList == null)

return;

Location location = locList[0];

dict.Clear();

var nav = location.Result.CreateNavigator();

XPathNodeIterator iterResults = nav.Select("All_Results/Result");

string myContent = "";

// concatenate document vectors

foreach (XPathNavigator res in iterResults)

{

var docVectorNode = res.SelectSingleNode("docvector");

if (null != docVectorNode)

myContent += docVectorNode.Value; // [term1, weight1]..[termN, weightN]

}

if (myContent.StartsWith("["))

{

// remove surrounding brackets, and split term and weight

myContent = myContent.Remove(0, 1);

myContent = myContent.Remove(myContent.Length - 1, 1);

var array = myContent.Split(new string[] { "][" }, StringSplitOptions.RemoveEmptyEntries);

for (int i = 0; i < array.Length; i++)

{

string[] keyvalue = array[i].Split(',');

string key = keyvalue[0];

string val = keyvalue[1];

if (dict.ContainsKey(key))

dict[key] = dict[key] + Double.Parse(val);

else

dict.Add(key, Double.Parse(val));

// only keep 10 docvector items for each result

if (i >= 10)

break;

}

base.OnPreRender(e);

}

protected override void RenderContents(HtmlTextWriter writer)

{

if (dict.Count > 0)

{

// order terms in dictionary by weight

var words = from k in dict.Keys

orderby dict[k] descending

select k;

int fontsize = 30;

string color = "3333CC";

int step = (30 - 8) / dict.Count;

writer.Write("<p><center>");

foreach(string word in words)

{

                    // output one term...
                    writer.Write("<a href=\"results.aspx?k=" + word + "\" title=\"" + word + "\" style=\"color:#" + color + ";font-size:" + fontsize + "pt\">" + word + "</a>    ");

                    // ...set smaller font for next term

fontsize = fontsize - step;

// ...and alternate color

if ("3333CC".Equals(color))

color = "9999FF";

else

color = "3333CC";

}

writer.Write("</center></p>");

base.RenderContents(writer);

}

Testing

To test the web part you need a FAST Search Center on your SharePoint installation, plus some content indexed and searchable. Build and deploy the Web Part by hitting F5 in Visual Studio, and you are ready to start testing (and debugging!). Add the new Web Part to the result page in your search center; Click Edit Pag > Click Add Web Part in e.g. Bottom Zone > Select TagCloudWebPart from Custom category > Click Publish. Execute a query, and you will see a tag cloud like in the image at the top of this blog post.

Summary

The key point with this exercise was to show how you can easily create new search Web Parts that play nicely together with other search Web Parts. With SharePoint 2010 this is done through the SharedQueryManager; here you can hook in to get the results, or hook in to manipulate the query. Either way, the programming model is the same for FAST Search and SharePoint Search (and OpenSearch). You work with the QueryManager and the Locations in the Federation OM.

Links

See links below for more resources;
- The WebPart class on MSDN
- Visual Studio 2010 support for SharePoint development

The two types of Search Service Applications (in a SharePoint 2010 deployment with FAST Search Server)

arnts — Wed, 27 Jan 2010 11:42:00 GMT

Let's look more in detail at a typical FAST Search Server for SharePoint 2010 deployment, and in particular at the Search Service Applications (SSAs) in such a deployment. SharePoint 2010 with FAST Search as the search provider will typically have two SSAs; the FAST Query SSA and the FAST Content SSA. Note that these two SSAs are differerent types of SSAs, i.e. of type Regular and ExtendedConnector respectively (useful to know if you want to script and automate setup).

The main reason for having two types of SSAs is the separation of content; People content is indexed and served by the SharePoint Server, while all other content (documents, sites, pages, etc) is indexed and served by FAST Search. Here's what their respective responsibilities are;

The FAST Query SSA has two main responsibilities: 1) serve all querying (against both FAST Search and SharePoint Search), and 2) crawling people content. This SSA will route people queries to SharePoint Search, and all other queries to the FAST Search query service. The queries are routed based on default search provider, a property on the SSA, or by explicitly setting a provider on each query request. Note that the FAST Search query service location is one of several FAST specific properties on the SSA.
The FAST Content SSA (aka Connector SSA) is responsible for crawling and feeding content to the FAST backend. It will be configured (in Central Admin) with all content sources, except the people content source which must be setup on the FAST Query SSA. The content SSA will know where to feed crawled documents through the extended connector properties (like where the FAST content distributor is).

The figure below gives an overview of where the SSAs fit in the search architecture.

You can of course use PowerShell to list and update settings on the SSAs, e.g.:

List all FAST properties on the FAST Query SSA:

> Get-SPEnterpriseSearchExtendedQueryProperty -SearchApplication "Search Service Application"

Set the default provider on the FAST Query SSA:

> Set-SPEnterpriseSearchServiceApplication -Identity "Search Service Application" -DefaultSearchProvider "FASTSearch"

You can find more details on the SSAs in the Deployment and configuration of FAST Search Server 2010 for SharePoint (Beta) documentation. More information on the Service Application Framework (SAF) can be found on MSDN, and in several blog posts here.

Consume the SharePoint Query OM from PowerShell

arnts — Wed, 02 Dec 2009 17:35:00 GMT

In my previous post I mentioned that you can use the same search object models and interfaces to query both FAST Search and SharePoint Search in SharePoint 2010. So let's look at how you can accomplish this by issuing a simple query using PowerShell.

First, start a PowerShell window by running "SharePoint 2010 Management Console" located in your Microsoft SharePoint 2010 Products folder on the start menu. This will automatically load the Microsoft.SharePoint.PowerShell snap-in so you can access SharePoint PowerShell commands. You can also access the Query OM classes from this PowerShell window, and the script below shows a simple example of how you can issue a query using the KeywordQuery class:

# setup a keyword query object

$site = New-Object Microsoft.SharePoint.SPSite "http://localhost"

$kq = New-Object Microsoft.office.Server.Search.Query.KeywordQuery $site

# set some query properties...

$kq.ResultTypes = [Microsoft.Office.Server.Search.Query.ResultType]::RelevantResults

$kq.RowLimit = 5

$kq.QueryText = "microsoft"

# issue the query...

$resultTableCollection = $kq.Execute()

# get the result table...

$relResultTable = $resultTableCollection.Item([Microsoft.Office.Server.Search.Query.ResultType]::RelevantResults)

$relDataTable = $relResultTable.Table

# and output the results

$relDataTable.Rows | Format-Table -AutoSize -Property Url,Title

As you can see, there is no explicit reference to which search provider (SharePoint Search or FAST Search) to use in this script. This means that the default provider on the Search Service Application (see architecture overview) will be used. So the same script can be run, regardless of search provider, to get search results..

If you want, you can explicitly set the provider. This is done by setting the ResultsProvider property on the KeywordQuery object before executing the query.

$kq.ResultsProvider = [Microsoft.Office.Server.Search.Query.SearchProvider]::FASTSearch

The Query OM provides a unified interface for querying both SharePoint Search and FAST search, while PowerShell provides a quick and easy way of accessing the OM. Simple scripting in Powershell can be useful in several cases, e.g. to

Test different query features. Create e.g. a script that queries with and witouht stemming, and compare results.
Test and validate query setup. To see e.g. if you configured your Search Service Application correctly for FAST Search, set the result provider to FASTSearch and issue a query. If setup is incomplete, you will see an exception

Search driven application development on SharePoint 2010

arnts — Fri, 27 Nov 2009 15:53:00 GMT

Last week I wrote a high level overview on The Search Developer Story in SharePoint 2010 over at the Enterprise Search Blog. Next, I'll provide a more detailed look at the different integration points in a series of blog posts. It will be aimed mainly at developers and architects wanting a better understanding of the technical architecture on the search side, and how to leverage the SharePoint platform for developing search driven applications.

Before diving into the details, let me provide a summary of the query side integration options. The diagram below shows the main query side integration points, applicable for both SharePoint Search and FAST Search for SharePoint, which now share a common platform and interfaces.

As shown in the figure, applications based on the SharePoint 2010 platform can leverage different query components (all available on the web front end in a SharePoint deployment). You can read more about the details of each integration point here, including benefits and some advanced features available with FAST Search only. For your convenience, here's a quick recap:

Search Web Parts

Shared for SharePoint Search and FAST Search, providing a unified user experience

Based on the Federation OM

The main SharePoint object models on the query side are the Federation OM and the Query OM.

The Federation Object Model (OM)

A new object model in SharePoint 2010

Provides a unified interface to search against different locations/engines

Allows for combining and merging of results

Supports 3 locations out-of-box in SharePoint 2010

SharePoint Search

FAST Search

OpenSearch

The Query Object Model (OM)

The lower level, typed object model

The object model that other services and object models are based on

Communicates with the Search Service Application (SSA) for getting search results

Integration points that also are available for applications outside SharePoint farm, e.g. standalone non-web based applications and Silverlight applications;

The Query Web Service

SOAP based ASMX service

Uses same schema for FAST Search and SharePoint Search

Provides search results, query suggestions, meta data and more

The Query RSS

Lightweight integration point

Coming up, more details on the Search Service Application, the object models, the web service and more. Stay tuned!

FAST Search Server for SharePoint 2010 Beta available for download

arnts — Fri, 20 Nov 2009 20:39:00 GMT

I wanted to kick off this blog with some links and resources related to last week's SharePoint 2010 beta announcements. In particular I wanted to focus on the search side, where there now are two main offerings; the out-of-the-box SharePoint Search, and the add-on FAST Search product. The FAST Search product is new for SharePoint 2010, and the Enterprise Search blog lists 10 reasons why you should consider it.

If you want to check out FAST Search for yourself you can just download SharePoint 2010 Beta and FAST Search for SharePoint Beta. You should also check out the Tech Center for more resources, and here are some additional links to get you started;

For an early review on SharePoint 2010, see e.g. the NetworkWorld article. The SharePoint blog also has a great overview of what's new, including on the search side.