Connecting the Dots : Visualize Organize & Search

Using ISearchQueryHelper from managed code

cheller — Thu, 07 Dec 2006 01:13:41 GMT

ISearchQueryHelper is a COM interface that can be used to convert from Advanced Query Syntax (AQS) to a SQL query that can be passed to the OLE DB Provider for Windows Search. (For background, see my previous post: Advanced Query Syntax: What, Where, Why, and How)

Good news for managed code developers - there's a type library from which you can create an interop assembly (using the tlbimp.exe tool). You'll find the type library (it's called SearchAPI.tlb) in the Lib folder of the Windows SDK.

Once you've generated an interop assembly and have added a reference to it from your managed code project, here are the necessary steps to convert from AQS to SQL syntax:

Instantiate the Search Manager
Get the catalog manager for the system index
Query the catalog manager for its corresponding query helper
Convert an AQS string to a SQL query string

Here's some C# code that shows how to convert an AQS string to a SQL query string (the code fragment assumes that aqsQuery is a string variable that contains an AQS query):

using SearchAPILib;

...

CSearchManager manager = new CSearchManagerClass();
CSearchCatalogManager catalogManager = manager.GetCatalog("SystemIndex");
CSearchQueryHelper queryHelper = catalogManager.GetQueryHelper();

string sqlQuery = queryHelper.GenerateSQLFromUserQuery(aqsQuery);

Advanced Query Syntax: What, Where, Why, and How

cheller — Tue, 05 Dec 2006 21:56:02 GMT

What is Advanced Query Syntax and where is it used?

Advanced Query Syntax (AQS) is a syntax that enables end-users to define targeted search queries, and is supported in a number of Microsoft products (for example: Windows Vista, Outlook 2007, and Windows Desktop Search).

Here is an example of a search query defined using AQS that will return all the email items, sent by me today, that include the word "Vista".

Vista author:catherine kind:email sent:today

For information on the specifics of the syntax, check out the documentation on MSDN.

Why should I care about AQS?

As a user (and developers are users too :-), AQS can help you find things faster on products that support AQS. For example, on Windows Vista with Outlook 2007, I can enter the above query directly in the Start menu's instant search box!

As a developer, if you're using the OLE DB Provider for Windows Search to run user-specified queries, you may want to consider supporting AQS.

How do I support AQS in my application?

There's no need to parse AQS yourself, just use the ISearchQueryHelper interface. This interface has a method called GenerateSQLFromUserQuery that converts AQS to a SQL query that can be sent to the OLE DB Provider for Windows Search.

Windows Vista Search: Syntax Update

cheller — Fri, 10 Nov 2006 18:35:42 GMT

I recently learned some new things about the Search APIs in Windows Vista, including some great improvements in the SQL syntax for the OLE DB Provider for Windows Search. (Thank you Chris McConnell - Architect on the Search team - for driving these changes, and for taking the time to explain them to me!).

There've been some simplifications in the syntax that happen to address a couple of things I found to be a bit odd and cumbersome:

System property names no longer require double quotes. Recall that the column names specified in the SELECT statement must be properties from the Windows Vista property system; and since the naming convention for properties includes dots (System.Application.Property), property names had to be delimited with double quotes. The double quotes are no longer necessary for system-defined properties (the ones that begin with System.)
..SCOPE() no longer required in FROM statement. There is only a single catalog that can be searched on the desktop - and since scope is actually defined using the "Scope=" predicate in the WHERE clause - you no longer need to add that funky "..SCOPE()" stuff at the end of the FROM clause.

So, a query like this:

SELECT "System.ItemName", "System.ItemUrl" FROM SYSTEMINDEX..SCOPE()

can now be written like this:

SELECT System.ItemName, System.ItemUrl FROM SYSTEMINDEX

Much better, isn't it?

(Note that these changes reflect added flexibility, and won't break any existing queries)

Windows Desktop Search 3.0 Released

cheller — Thu, 26 Oct 2006 02:17:24 GMT

Windows Desktop Search (WDS) 3.0 has been released to the web. You can download it from one of the following links (depending on your OS version):

Why doesn't Windows Vista appear in this list? Because the stuff that ships in WDS 3.0 is already built-in to Windows Vista - there's no need to install it.

So if you're targeting Windows Vista (which I hope you are :-), why should you care? You can think of WDS 3.0 as a "redistributable" for the Windows Vista "Visualize, Search & Organize" features that go down-level (e.g. the indexing platform, the property system, the OLE DB Provider for Windows Search). This enables you to target these features and still run on Windows XP, with the addition of WDS 3.0. Examples of applications that do this are Outlook 2007 and OneNote 2007.

Windows Vista Search Syntax: WHERE are my predicates?

cheller — Wed, 28 Jun 2006 02:44:00 GMT

In my last post, I gave a quick overview of the query syntax for the OLE DB Provider for Windows Search that ships in Windows Vista. In this post, I'll go over some of the details of the predicates that can be used in the WHERE clause.

Just a quick disclaimer: this post is meant to help you get started; it's not my intention to provide a complete reference of the entire syntax.

Simple Predicates

Simple predicates include LIKE, literal comparisons, and the NULL comparison.

Here are examples of the WHERE clause, using simple predicates:

WHERE ~~"System.DisplayName"~~ "System.ItemNameDisplay" LIKE 'a%'
Returns all items whose name begins with the letter 'a'

[Updated on October 25, 2006 to reflect changes made to the property system since I wrote this post. The property System.DisplayName no longer exists; use System.ItemNameDisplay instead]

WHERE "System.DateModified" >= '2006-06-01'
Returns all items modified on or after June 1^st, 2006

WHERE "System.ApplicationName" IS NOT NULL
Returns all items that have a value for System.ApplicationName

Full-Text Predicates

The full-text predicates, CONTAINS and FREETEXT, enable you to search for text within the contents and/or the properties of items.

They both have a similar syntax:

CONTAINS | FREETEXT ('<searchString>') searches within the contents of items

CONTAINS | FREETEXT (<column>, '<searchString>') searches within a given property

CONTAINS | FREETEXT (* , '<searchString>') searches contents and all properties

Here are some simple examples, using different combinations of the CONTAINS predicate:

CONTAINS(*, 'Catherine')
will return all items that contain 'Catherine' in either its contents or in any of its properties.

CONTAINS("System.Author", 'Catherine')
will limit the search to the property specified as the first argument – in this case, System.Author.

CONTAINS('Catherine')
will limit the search to the item contents.

Note the following:

The predicate argument is enclosed in parenthesis
The search string is always enclosed in single quotes
As we saw in the SELECT statement, column names are enclosed in double quotes
Searches are case-insensitive

Regarding the format of <searchString>, if you want to search for anything beyond a single word with no spaces, you will also need to use double quotes. This is in addition to the single quotes that enclose the entire condition. Here are some examples:

CONTAINS(' "This is a phrase" ')
note that the search phrase is within double quotes, and the entire argument within single quotes

CONTAINS(' "Using a wildcard*" ')
uses * as a wildcard in the search string

CONTAINS(' "Using" AND "Boolean" OR "operators" ')
will return items that contain the string 'using' and the string 'boolean' or the string 'operators'.

CONTAINS(' "This" NEAR "that" ')
will return items that contain the string 'this' within approximately 50 words or less of the string 'that'.

Use FREETEXT if you care about relevancy– it’s the only predicate that returns a meaningful rank value. (All other predicates return 0 or 1000, depending on the whether there was a match or not). (Hint: to get the rank value, include System.Search.Rank in the select list)

Search Depth Predicates

Not sure what the official term for this group of predicates is, but the SCOPE and DIRECTORY predicates enable you to control the depth of the search.

The SCOPE predicate will search a specific folder and all of its subfolders (sometimes referred to as deep traversal).
The DIRECTORY predicate will search a specific folder, without searching subfolders (or what’s sometimes referred to as a shallow traversal).

The syntax for these two predicates is the same:

"SCOPE | DIRECTORY" = '(file | mapi | csc):<path>'

Here’s an example of a SCOPE predicate that limits the search to the C:\Foo\Example folder:

"SCOPE"='file:C:/Foo/Example'

To only search the root folder, here’s what the DIRECTORY predicate would look like:

"DIRECTORY"='file:C:/Foo/Example'

The syntax may seem a bit odd (or at least, it did to me); let me point out some of the details:

"SCOPE"=
- The predicate is enclosed in double quotation marks (it’s case insensitive, so it doesn’t matter if you use upper or lower case) and followed by an equal sign
'file:C:/Foo/Example'
- The path string is enclosed in single quotation marks
- The path string begins with the corresponding protocol (in this case, file) followed by a colon; it's not followed by the familiar //, as in URLs.
- Forward slashes (/) are used in the path

A Couple of Closing Hints

The Windows Vista search and indexing infrastructure shares a common lineage with the SharePoint Portal Server indexing technologies. I've found the SharePoint Portal Server Search SQL Syntax documentation to be quite useful in piecing together some of the syntax details. Just keep in mind that not everything is applicable to Windows Vista.

Regarding the Windows Vista property system - there have been recent changes to some of the property names to improve clarity, so you may need to change your queries to run with post-Beta 2 releases.

Searching for Windows Vista Search?

cheller — Thu, 22 Jun 2006 02:06:00 GMT

A number of people have been asking about Windows Vista search and how to programmatically query the search engine. There's not a lot of documentation available yet (and this will obviously change as we near the release of Windows Vista). So in the meantime, here's a quick primer for those of you that would like to get started today.

Background
Windows Vista includes a built-in desktop indexing platform. You’ll see search integrated throughout Windows Vista, in all the Explorer windows and even in the Start menu. A number of Microsoft products are building on top of this infrastructure - like Outlook 2007 and OneNote 2007, just to name a couple.

Your applications can also plug into this same infrastructure and query the index, by using the new OLE DB Provider for Windows Search. (There are also extensibility mechanisms by which you can expose data to the indexer, but I'll leave that for a future post)

The Windows Vista search and indexing infrastructure will also be available as a download for Windows XP and Windows Server 2003. You can download the Windows Desktop Search 3.0 Beta Engine Preview here.

Using the OLE DB Provider for Windows Search
While there's a lot of information available on OLE DB, you won't get very far with that alone – you also need to know the specifics of the provider, like its connection string and its query syntax. (If you're using ADO.NET, just a reminder that you can use the types defined in the System.Data.OleDb namespace. If you don't know how to use an OLE DB provider, a good place to start would be the MSDN Data Access and Storage Developer Center.)

Here's the connection string for the OLE DB Provider for Windows Search:
"Provider=Search.CollatorDSO;Extended Properties='Application=Windows';"

The query syntax is what I've found to be the trickiest part, and that's what I'll focus on for the rest of this post.

Statements
The OLE DB Provider for Windows Search supports a single statement: the SELECT statement. The provider was designed solely for read-only operations, so there's no need for INSERT, UPDATE or DELETE statements.

The structure of the SELECT statement looks like this:
SELECT <properties>
FROM [machineName.]SYSTEMINDEX..SCOPE()
[WHERE <predicates>]

SELECT <properties>
<properties> represents a list of one or more comma separated "column" names, where the columns correspond to properties defined in the new Windows Vista property system.

A note about the new Windows Vista property system: Windows Vista has a new schema-based property system which defines the metadata that can be stored within files. Hundreds of system-defined properties will ship with Windows Vista and the property system is also extensible by custom file format providers.

The documentation for Property Description Schema in the SDK docs will give you a pretty good idea of how properties are defined.
Documentation on the system-defined properties, while not yet available online, is available in the documentation installed with the Beta 2 release of the Windows SDK. Here's how to navigate to the docs for these properties: User Interface->Windows Shell->Shell Reference->Shell Properties

Here are a couple of details about the SELECT clause that may not be obvious:

The convention used for property names is: Publisher.Application.Property
Property names must be enclosed in double quotation marks (due to the period used in the naming convention)
SELECT * is not supported, so you'll need to specify at least one property name

FROM [machineName.]SYSTEMINDEX..SCOPE()
The FROM clause is pretty straightforward; since there's only a single index that can be queried, there's only one possible variation: you can precede SYSTEMINDEX..SCOPE() with a machine name to execute a query against the local index of a remote machine. (You must be running Windows Vista or Longhorn Server on the remote machine, and there are configuration steps/permissions that must be done before this will work).

[WHERE <predicates>]
The optional WHERE clause supports a number of predicates.

Simple predicates: literal value comparisons (<,>,=) and LIKE
Full-text predicates: CONTAINS and FREETEXT
Search depth predicates: SCOPE and DIRECTORY

In my next post, I'll cover some details about the predicates and I'll provide some specific examples of their usage.

Updated June 26, 2006: modified formatting (you should now be able to copy and paste the connection string into your code, without getting a runtime error from the "smart" quotes)