Mark Arend : Search

Colleagues, Social Distance & Relevance in People Search; Social Networking tools

markarend — Thu, 01 May 2008 16:16:00 GMT

MOSS provides some very powerful features to enable the new buzzword "social networking." Discussions about these can be found pretty easily, and I give an overview of some tools near the end of this posting. But specifics about how some of the details work has been devilishly hard to find... until now. Through discussions with various people in-the-know, I've been able to assemble answers to common questions around People Search and MySite web parts.

Social Distance

By default, the results on the People Search page are ordered by Social Distance. What the heck is that? It's an ordering of results based on colleague relationships. Here's how it's computed:

Search results are always returned to the search results page sorted by relevance.
When viewing results by social distance, additional processing on the search results page is used to group the results:
- The first 3 pages of search results are grouped by colleague-ness: first your colleagues appear, then colleagues of your colleagues, then everyone else.
- Within each group, the ordering is still by relevance.
- When paging through results, another 3 pages of results will be grouped once you reach page 4, then page 7, etc.

By default, 10 results appear per page, so this groups people in batches of 30. If you customized the People Search results page to show 25 results per page, this would group people in batches of 75.

There are potentially some quirks with this algorithm that was designed to give the best value in a lightweight manner. Your colleagues will appear at the top of the list on page 1, but if you have lots of colleagues, more of them might appear on page 4. This is because the colleague grouping takes place on the client, starting with a relevance ranking. So if some of your colleagues are relevance-ranked after 3 pages worth of results, these will be grouped once you get to the next set of 3 pages (page 4, page 7, etc). Based on relevance, these people should be less relevant to your query.

The People Search results page allows you to toggle the results ordering to only Relevance, ignoring the Social Distance grouping.

Relevance

Relevance ranking in people search is the same as that used in regular search across lists and documents. I'm told that the ranking algorithm MOSS uses in all cases is BM25F.

Conceptually, a user's profile is treated as a document about that person with their name as the title. At a basic level, the more search query terms that appear in the person's profile, the more relevant the result. Matches against a few special fields (e.g. name, alias) are returned in the high confidence webpart.

Note that the names of discussion lists that a user belongs to are included in that user's profile information. This means that a people search for "rock climbers" should return all of the people who are members of a distribution list named "Rock Climbers at Contoso" as well as anyone who has the phrase "rock climbers" in their profile information, such as in their "About Me" field.

Colleagues

Here's where the real value of social distance and other meaningful relationships is computed.

A list of your colleagues is stored in the profile database. During a profile import, this list is initially populated with a set of "immediate colleagues" that are computed from your profile properties: manager, peers and direct reports. This way, when you first look at your Colleagues web part, it's not blank.

Then, from an editing page on your Profile, you can edit your colleagues list to add & remove specific people, and to group them and set their visibility to other people. Also from here, you can ask SharePoint to suggest other colleagues, and these suggestions are collected from several places.

My Microsoft badge photo, 1999!

From your MySite page (or your Profile page, which is available even if MySites are turned off), click Colleagues to edit your colleague list.

Where do colleagues come from?

Immediate colleagues
...automatic: during profile import; comprised of:
- Your manager
- Your peers (others who report to your manager)
- Your direct reports
Colleagues added by you
...manual: via editing page on your Profile
Suggested colleagues
...lookup: manually guided via editing page on your Profile; comprised of:
- Sent items in Outlook. Periodically (every 5 days or so, depending on usage) Outlook analyzes Sent Items for common recipients, weighted by frequency of contacts and other factors. This is how Outlook is able to suggest likely recipients when you begin typing their name in the To: line, and in my experience, it seems to work pretty well. When you click Suggested colleagues, it collects these names from Outlook via an ActiveX control, and adds them to the list of suggestions.
- Office Communicator contacts are examined; that is, those contacts that you've added to your Office Communicator client from the company directory. These are also collected from the Communicator client via an ActiveX control.
- Site Memberships are all sites in which you are explicitly included in the "~Members" group for the site (where ~ is your site name). Other users who are also explicitly included in the ~Members group on those sites are suggested as your colleagues. This information is collected by a WSS profile synchronization timer job.
- DG (Distribution Group) Memberships are examined for suggestions. Distribution groups stored in AD are collected during the AD import.

Colleagues from these sources are collected and the top 20 or so are displayed as suggestions when you click "Add Colleague" on your colleague editing page. It's not fruitful to pursue a more detailed understanding of this calculation, because the recommendation for improving the "quality" of colleagues found is the same: people associated with items you access more often are more likely to be suggested as your colleagues. If you don't like the suggestions, you can remove individuals from the suggestion list one by one or all at once from this page, then get a new list of suggestions.

Privacy

Whoa! Is SharePoint looking at my email? I don't want that, how do I prevent it?

First, SharePoint's not looking at your emails. It gets information about who you sent emails to by asking Outlook for that information. Second, it will never do this without asking your permission. You can always say no:

If you say Yes, then SharePoint will assemble a list of suggested colleagues for you, and you can accept or reject these suggestions. But even if you say No, you can still edit your colleagues to search for and add new people, and to set the privacy of who should see that they are your colleague (Everyone, My Colleagues, My Workgroup, My Manager and Only Me):

Social Networking tools

Why did we go through all this trouble to try and figure out who your colleagues are? Social Networking, baby! We spent a good bit of time on the inputs and the outputs of this to make it a really useful function, instead of just paying lip service to it.

The inputs are the information about who your colleagues really are. A simple option would have been to just provide a place for you to register your colleagues, and not go through all this supposedly intelligent guesswork. But that would have made it almost useless, because most people are just not going to take the time to populate this list, and even fewer will maintain it as their colleagues change over time, which is natural.

SharePoint uses the information already embedded in your company's infrastructure and the changing data you deal with on a daily basis to help you connect with the right people.

Sounds like market-speak, but that's really the goal and I think it does an amazing job.

The outputs are the tools that SharePoint can provide based on this colleague information. Consider some of these details:

People Search ranking by Social Distance. By default, the top people returned when you search for a name will be the people you most likely want to find: people you've communicated with recently, perhaps as a participant on an email thread or a collaborator on a site, and now you want to find more information about them.
In Common with You web part on My Profile page. When you view someone else's profile (for example, by clicking their People Search result), this web part shows you a list of people that you both may have in common. This is useful to quickly understand business relationships that you may need to know about. Categories shown are
- Manager we both report under
- Colleagues we both know
- Memberships we both share
Colleague Tracker web part on MySite. This shows you recent changes to user properties of your colleagues, effectively a newsletter of what's new with people you work with. Information you can track includes
- Anniversaries
- Profile property changes
- Membership changes
- New documents this person has posted
- Out of Office status
- Blog postings from this person
Colleagues web part on My Profile page. This displays your current list of colleagues, grouped into categories that you may have applied.
Memberships web part on MySite. This shows Distribution Lists and SharePoint Sites of which you are an explicit member. For SharePoint Sites, it is rather specific: you must appear as an individual member of the ~Members group on the site (where ~ is the site name). If you have access to the site through a different configuration, such as being a member of the ~Owners group, or being a member of an Active Directory Security Group that is a member of the ~Members group, this is not counted. Otherwise, everyone would very likely see all the portal sites displayed as their memberships, and that would not be useful.
Organization Hierarchy web part on My Profile page. This shows your manager, your peers (others who report to your manager) and your direct reports, if you have any.

These are some of the tools that MOSS provides for Social Networking; I've highlighted the ones that use profile data and the powerful "colleague" pattern to show meaningful relationships among people. For a better introduction to the full set, see the product team's Enabling and Managing Social Networks for Business use with MOSS.

Technorati Tags: SharePoint,Social Networking,MySites,People Search,Social Distance

MOSS Search UI Architecture - Introduction

markarend — Fri, 06 Jul 2007 23:13:00 GMT

To modify SharePoint’s default search behaviors, it’s important to understand how the user interface elements for search work together to specify a query, execute a search and display the results. The following high-level overview shows how searches are executed and displayed.

1. The search box near the upper right of most pages is a control that accepts search keywords and scope from the user, and passes them to the search results page using a query string and HTTP GET.

2. The search controls on the advanced search page accept keywords, scopes, languages and properties from the user, and passes them to the search results page using form variables and HTTP POST.

3. The search results page calls an internal class with the inputs it receives, which builds the query, issues the query against the index and receives the results. The internal class is implemented as a singleton: Microsoft.Office.Server.Search.WebControls.SearchResultHiddenObject

4. Web parts on the search results page retrieve their portion of the results from the internal class to render their specialty: core results, high confidence results, page number, summary or statistics.

This architecture provides a very powerful and flexible way to configure different types of search results. Unfortunately it is restricted to Keyword Syntax queries and is not possible to take advantage of the search engine's ability to use SQL Syntax queries, without development effort. The development effort could be rather significant, because the internal class that performs the keyword syntax search is not available for custom class derivation.

For the form fields used from the Advanced Search page, here's a table of form fields that I came across. It was constructed using reflector and reverse engineering of search and search result pages.

Forms Param	Used for	Legal values and comments
ASB_OT_	?	?
ASB_TQS_AndQ_tb	AND query text	Url encoded string
ASB_TQS_PhraseQ_tb	Phrase query text	Url encoded string
ASB_TQS_OrQ_tb	OR query text	Url encoded string
ASB_TQS_NotQ_tb	NOT query text	Url encoded string
ASB_SS_scb_<index>_<scopeidnumber>	Site Scope	<index> is the selected index number value (0 based) <scopeidnumber> is an integer value identifier the selected scope When submitted the selected properties are included and the values are "on". If a property is not used then the form data is not included.
ASB_SS_lcb_<number>_<lanuagecode>	Language scope	<number> is index number of language (0 based) <languagecode> is an know number identifier for language. All the language codes can be found in the Properties property of the advanced search web part. Forms data is not there if none are selected
ASB_SS_rtlb	Document type scope	Legal values from adv search box are: "default "= all "documents"=Documents "worddocuments"=Word documents "exceldocuments"=Excel documents "presentations"=Presentations
ASB_PS_plb_<propnumber>	Property name	<propnumber> runs from 0 to 5 Contains "(pick property)" if there is no content
ASB_PS_olb_<propnumber>	Property operator	<propnumber> runs from 0 to 5 Contains "contains" if there is no content
ASB_PS_pvtb_<propnumber>	Property value	<propnumber> runs from 0 to 5 Is null if there is no content
ASB_PS_lolb_<propnumber>	Property operator (between expressions)	And/Or boolean expressions expected Form data is included if the property expression is followed by a new property expression
ASB_TextDT_Props	Text properties	#;# seperated field
ASB_DateTimeDT_Props	DateTime properties	#;# seperated field
ASB_ResType_Query	Type of search	FullTextQuery, if form data is available otherwise Keyword query
ASB_BS_SRCH_1	The search button	Search

MOSS Search Query Syntax - Keyword, SQL and URL

markarend — Fri, 06 Jul 2007 23:05:00 GMT

MOSS 2007's search engine supports three different types of query syntax: Keyword, SQL and URL.

MSDN article Building Windows SharePoint Services Search Queries gives a reference for these syntaxes. What it doesn’t make clear is that MOSS's out-of-box Search UI only exposes Keyword Syntax, and that custom development is needed to utilize the other syntaxes.

On a related note, you can use SQL Syntax for the ContentQuery web part by specifying your query in CAML. MSDN article How to: Customize the Content Query Web Part by using Custom Properties discusses this.

Search Results Page may be different per Search Scope

markarend — Wed, 24 Jan 2007 17:38:00 GMT

Microsoft Office SharePoint Server 2007 provides two categories of search scopes: Custom and Contextual.

Custom scopes:

“All Sites”
“People”
The custom search scopes created via the search or site admin pages

Contextual scopes:

“This Site”
“This List”

Issue

Search behaves slightly differently between these categories. Specifically, there are two pages on which search results are displayed, one for each category. The page for Custom scope results is the most familiar: it’s the one that offers custom scopes and properties available via Advanced Search. The page for Contextual scope results is much simpler: there are no best bets, high confidence results, advanced search link nor search tabs. This page is also customizable, but is done at the level of a web designer or farm administrator, not in the same way as the one for Custom scopes, which can usually be customized pretty easily by individual site administrators.

Also, the search results page that can be configured on the Search Settings page affects Custom scope results, and defaults to /SearchCenter/Pages. This setting has no effect on Contextual scope results, which always go to /_layouts/osssearchresults.aspx for MOSS installations (for WSS-only, it is /_layouts/searchresults.aspx).

Impact

When people use the “This Site” or “This List” scope to search, the search engine will perform as it always does, but the results will be shown on a different page and the users may be confused by missing functionality on that page.

“Justification”

The two scope categories are implemented by two different products: contextual scopes are provided by the WSS platform, while custom scopes are provided by the MOSS product running on that platform. This is not an excuse, merely an explanation. It could be argued that it's not so bad for these results pages to differ when someone is searching This Site or This List, because their focus is narrower and they don’t need the advanced results as they do when searching much broader scopes. But users naturally express confusion over why the results are organized differently.

Workaround

Jan Geisbaur commented that he found a pretty simple workaround to redirect to a custom search results page:

1. Create a new aspx through sharepoint designer (named custom-search.aspx)
2. Copy code from osssearchresults.aspx to custom-search.aspx, and modify the masterpage
in custom-search.aspx
3. make a backup of osssearchresults.aspx and replace code with the following javascript:

As you can see this takes the parameters from the search box and passes it to custom-search.aspx

Remember when you make the special osssearchresults.aspx file, it is unghosted. This means you should create a feature that will install this unghosted file to current site collections, enabling you to deploy it.

Other Options

The following were original ideas for changing this behavior to make it more consistent.

Create a new search box control that builds a list of scopes that know how to redirect contextual scopes to the same results page as Custom scopes. Note that it will probably be easier to build a control that does not inherit from the search box control, because the job of this control is very simple and it would be extra effort to understand and override the base behavior. Not an option; the result page is hardcoded and must be overridden after the query.
Convert the contextual search results page to a redirect to the custom search results page. The key to understanding this is that the actual search is executed by the SearchCoreResults control, not the SearchBox control, so it's just a matter of making sure the search term data gets to the right results page. This option is described above as the workaround.
Customize the contextual search results page to work more like the custom search results page. Challenge here is this page is more difficult to configure, and it may not be possible to fully match functionality. This is not really an attractive option.
Disable custom scopes or contextual scopes. Using one of these approaches is the fastest way to eliminate confusion, but also results in limited functionality.
Educate users on difference of functionality. Part of this could be a blurb appearing on the simple contextual search results page indicating that more options are available when searching across wider scopes.

Workaround for broken search controls in B2TR

markarend — Fri, 13 Oct 2006 18:55:00 GMT

Since upgrading from B2 to B2TR, many people have reported that their search scope dropdown box now only shows a single scope for the current site, and that the Advanced Search link has disappeared. Some people have fixed this by creating new sites or site collections, but that didn't work for me. I stumbled across the following workaround, though.

On the Search Settings page, change the Search Results URL to use an "alternate location" instead of the default. You can use the same link as default, just need to enter it as the alternate location:

Just making this change restored my search controls to their full function! I put it back to default, and the control was again broken in the same way. Tried this on my customer's sites, and it worked there too.

There is another interesting difference between these two configurations. While the default choice for Search Results URL says it will present results in the default Search Center, showing the correct URL, this is is not where my search results are presented from when choosing this option. I get search results from /_layouts/OSSSearchResults.aspx page, which is not in the search center and not customizable via the SharePoint UI. Therefore it seems like this is the feature that's broken, the search control is merely responding to that.

I'd rather see my results in the SearchCenter, because I can customize that page easier than anything in the /_layouts directory, so this workaround solves both of these problems.

Searching with Property Filters

markarend — Tue, 10 Oct 2006 22:09:00 GMT

SharePoint can search for content using the values in site columns, as well as performing free text searches. The official help page that describes this is at http://msdn2.microsoft.com/en-us/library/ms476494.aspx. This posting gives some more examples of ways in which this is useful.

The names of the property filters you can use are the names of the columns on your document library or content type. One of the most useful property filters is ContentType, because with this you can search for specific types of documents; many values can be specified. There are a hierarchy of content types available, and specifying one will include all of its descendants. Here is a partial tree of the most pertinent values you can use for ContentType:

Page

Article Page

Welcome Page

Item

Document

Picture

Report

Custom document types

Announcement

Contact

Event

Issue

Link

Task

Folder

Discussion

Examples; Unions and Intersections

Search syntax does not provide Boolean operators AND & OR. But very similar functionality can be gained with the right combinations of properties. When one property is specified several times, all of the conditions are combined using OR. When different properties are specified, all of the conditions are combined using AND.

ContentType:"Announcement" ContentType:"Event"

Every list item that’s an Announcement OR an Event

ContentType:"Document"

All documents

ContentType:"Document" Author:"Mark Arend" Status:"Not Started" Status:"Draft"

All Documents created by Mark Arend that have a status of Not Started OR Draft

ContentType:"Article Page" -Status:"Published"

All Article Pages that are NOT Published. note the dash character to negate a property filter.

Other Filters

Property filters provide you with a way to narrow the focus of the keyword search based on the following:

Managed properties <property name>:<value>
Scopes <scope name>:<value>
Collapsed results <collapsed results type>:<value>

There is no maximum length for keywords property filters, however, the full length of the keyword query should be less than 1024 characters.

Keyword syntax supports querying for any properties of a string or numeric type from the full set of managed properties.

Name	Description	Examples
site	The URL property. Note: If a URL folder name contains a space, it must be enclosed in double quotation marks ("…"). Do not end a URL folder name with a trailing slash ("/").	site:http://employeesstg.amr.
scope	Compound scope. Specify the friendly name of scope.	scope:"Marketing Content"
duplicate	Collapsing filter. Requests duplicate items for specified URL.	duplicate:http://<displayUrl>

If there are one or more spaces in the display name of the column you want to filter by, then you must use the internal name of the column instead. Usually, this means replacing all spaces in the name with _x0020_. For example, to filter on Issue Status, use Issue_x0020_Status. Sometimes the space is just removed; for example, to filter on Contact Type, use ContactType. In some cases neither rule applies and the internal name is different; use a tool such as an object model browser such as SharePoint Inspector (http://www.codeplex.com/spi) to find the internal name (called the Static Name) that you need to use. Here are a few common ones:

Display Name	Static (internal) Name
Approval Status	ModerationStatus
Approver Comments	ModerationComments
Article Date	ArticleStartDate
Checked Out To	CheckoutUser
Collect Feedback	CollectF
Collect Signatures	CollectS
Contact E-Mail Address	PublishingContactEmail
Contact Name	PublishingContactName
Contact Picture	PublishingContactPicture
Content Type	ContentType
Created	Created_x0020_Date
Encoded Absolute URL	EncodedAbsUrl
Image Caption	PublishingImageCaption
Modified	Last_x0020_Modified
Page Content	PublishingPageContent
Page Image	PublishingPageImage
Page Layout	PublishingPageLayout
Path	FileDirRef
Rollup Image	PublishingRollupImage
Scheduling End Date	PublishingEndDate
Scheduling Start Date	PublishingStartDate
Server Relative URL	ServerUrl
Target Audiences	Audience
URL Path	FileRef

Very quick intro to portal/search concepts for SPS 2003 people

markarend — Thu, 13 Jul 2006 02:34:00 GMT

The term “Area” is no longer used in MOSS 2007. Areas were implemented as subsites, however, and it is still possible to create subsites on the Portal. The ability to manage the hierarchy and navigation of subsites replaces the ability to reorganize Topic Areas on 2003. Navigation links to subsites now may appear as tabs and as left-column links, mirroring the old Topic navigation methods, but are more flexible in what can be displayed where and in what position.

In MOSS 2007, you still have the concept of content sources and search scopes, but they are both more flexible. But you don’t administer them from the portal. You must go to the shared service provider administration page to create content sources. To do this, go to the central administration pages then click Shared Services Provider link in the left nav column. Content sources are lists of URLs at which search crawls are to begin; you can specify your entire site, or a series of special subsites on the portal that you want to search. You must do this to ensure the content on your subsites makes it into the index. Then, search scopes specify what portions of that index to retrieve results from. Scopes can be defined by what content source put them there, or by URL path with wildcard, or by filtering against a value such as contenttype or author.

Set these both up on the shared services provider administration pages, then go to your portal and choose which source scope(s) you want to include in your search control.