Engineering Windows 7

Welcome to our blog dedicated to the engineering of Microsoft Windows 7

Federating Windows Search with Enterprise Data Sources

Federating Windows Search with Enterprise Data Sources

  • Comments 60

The Windows Explorer has evolved by enabling you to find all sorts of content by searching for it. Many of you have used the search features in Windows Vista (based on our instrumented data) from the start menu or from the search box in Explorer. It has been a long time since most of us could remember where everything is by carefully managing our folder hierarchy and finding things based on file name alone.  We often rely on domain specific search (in music players, mail clients, photo clients) but with Windows Vista and Windows 7 we make it possible to search within a namespace and across namespaces.  This post is about a new feature based on Search that allows searching across PCs and even servers in an Enterprise setting.  Alwin and Scott, program managers, and Brandon, a developer, on the “Find and Organize” feature team authored this post. --Steven

Finding your stuff

Whether you’re searching or browsing, Windows Explorer is really about finding your stuff, and once you’ve found it, doing something with it (such as copying, opening, deleting, etc). For data that lives on your PC or home network, Windows 7 has invested in HomeGroup and Libraries (subjects for a future posting from our team) to provide an easier and richer experience than ever before. However, we didn’t stop there. Over the last few years, we’ve seen enterprise customers’ important content migrate towards (or aggregated in) centralized content stores, such as SharePoint. These products typically provide great features for team collaboration, document versioning and workflow management, archiving, retention policy enforcement, and other centrally-managed functionality that IT managers appreciate.

Where Enterprise Data Lives

Important enterprise data is found on local machines, in a variety of centralized content stores and also beyond the firewall

Unfortunately, this has placed an extra burden on customers to learn each new content store’s user interface, often asking them to give up familiar desktop features like drag-and-drop. Given their collaborative focus, these sites grow organically and it can become hard to remember where a particular document was stored and then wade through long lists of them every time you want to get back to it. Enterprise customers have asked us for a solution that simplifies finding important content in these various data stores but without leaving their normal Windows work flows.

As we looked at this trend and the lack of integration with content management and content indexing web services, we used these guiding principles in developing a solution:

  • Natural for people to use. Customers want a more consistent experience for finding and working with data in these disparate content stores, and would like us to bridge the local and remote content experiences by helping them “roll over” from one to the other.
  • Easy for IT admins to deploy. IT admins don’t like to deploy code, and want low-maintenance solutions that are easy to manage. Meanwhile customers want to connect up these sources without going through long and tedious installation processes or having to get help every time they want to set up a new search location.
  • Easy for developers to adopt. Developers want to enable this functionality in their offerings quickly and easily. There are a lot of data sources which need to be supported because IT folks don’t want to be locked in to a specific server technology.

Choosing to build Federated Search

Federated Search wasn’t the only way to address these challenges. The brute force approach would have been to take our existing Windows Search indexing technology and just use it on these content stores—that index the remote content on a local PC. This isn’t a very realistic solution since it’s inefficient to have all content indexed over the network by each person’s machine, especially when the content is changing at a rapid pace and represents a large corpus. Corporate retention policies may also prevent keeping even a local index of certain sensitive data.

Fortunately, there’s a better option – Federated Search. Federated Search enables you to search a remote web service from Windows explorer and get results back that you can act on like any normal file. The largest barrier to doing Federated Search has already been taken care of too. That is, most of these content stores are already indexed on the server, or at least on some server. There are several great offerings that will accomplish this, such as Microsoft Search Server. Not only do these servers index this content, but many of them already expose search results via a standard web protocol. This is largely thanks to the prevalence of OpenSearch and RSS enabled clients (including Internet Explorer and Microsoft Search Server, among many others).

For Windows 7, we’ve added support for Federated Search using OpenSearch v1.1 and worked to make the experience a seamless one. We found this solution strikes a good balance by leveraging the strengths of content services and the strengths of local file interactions within Windows.

Natural to use

Using Windows Explorer, people are familiar with several important user interface and interaction elements. They know how to use the navigation pane to change what they’re looking at. They know how to scroll around, how to select an item (or several), and they know how to double-click to open them. Most people know how to right-click for context-sensitive options related to their selection, or how to find those options presented in the command bar. They know they can drag and drop items to move them around. They know how to change view modes. We hope that they know how to search their current location using the search box, and in Windows 7 we think we’ve made it much easier to discover and use the Preview Pane to make sure they’ve got the right result.

Searching Sharepoint from Windows Explorer in Windows 7

Searching a SharePoint site using the new Federated Search support in Windows Explorer

Much of the usefulness of building Federated Search into Explorer is our ability to take advantage of this knowledge and familiarity. This may seem obvious once you see it in action, but behind the scenes there’s quite a lot going on to make all of this happen. For example, some applications such as Microsoft Word already know how to work with web URLs. So opening a Word document from a web server is fairly straightforward. But the majority of applications you’ll encounter really only understand how to open files on the local machine or via standard network file sharing protocols. This includes everything from the built-in software like Notepad and Paint, to third-party software like Photoshop or iTunes.

To handle this case, we implemented a “just in time” download solution, which will download the file to the internet cache before opening an application or taking actions (like using the SendTo menu) which require local files. This lets us offer searches that are very “lightweight” from a server load perspective, where we display metadata and icons or thumbnails without ever requesting the actual file. Then if you take an action like previewing or opening an item, we will do some behind-the-scenes work to make a local copy of the file only if necessary.

That enables us to work with the existing application ecosystem without asking anything of developers. However, applications can also take steps to offer even better functionality in many cases. For example, Windows Photo Viewer has added support for non-file items. So if you open a picture result in the built-in photo viewer, it’s the photo viewer that downloads the item, not Explorer. This may not seem like a big deal, but it lets the photo viewer enable the forward and back buttons to jump to the next or previous result – and it will download that image on-demand. Starting at the PDC we began reaching out to third-party ISVs to encourage them to implement similar enhancements for Federated Search scenarios, and we will continue to offer guidance on how to best integrate with all of the newest Explorer features.

Finally, we support all the standard clipboard and drag-and-drop operations. So if you drag a Word document from a Federated Search query onto your desktop, it will be copied there. You’ll even see the familiar Windows Explorer copy dialog, with progress indication, cancel ability, conflict resolution, and so on.

But wait, there’s more! Windows Explorer is a great tool that many customers know and love. But some people use it without even knowing it. Countless Windows applications make use of what we call the Common File Dialog. This is a special Explorer window that lets you find and choose items to be opened or inserted into your current application, without ever leaving it. If you’ve ever clicked File and then Open or Save in an application menu, you’ve probably seen some version of this dialog. PowerPoint, for example, uses the common file dialog to insert pictures. That means from inside PowerPoint you can click Insert Picture, select the Federated Search link for your image repository, search for the picture you want, and then insert it directly into PowerPoint. This works for any existing application that supports the Common File Dialog, and there are a whole lot of them!

Inserting a picture into PowerPoint’s using Federated Search

Inserting a picture into PowerPoint’s using Federated Search

Our Federated Search solution is all about simple lightweight access with a common, familiar user interface. This has a lot of benefits as we described above, but there are also cases where a server’s web interface will offer its own benefits. This might involve advanced query building, browsing, or server work-flow tasks, for example. So Windows 7 builds a bridge to these content repositories. After doing a search against a supported location, you will see a “Search on Website” button in the command bar which allows you to seamlessly send the query up to the service’s web interface in the default web browser. You’ll also see the “Open File Location” menu item when you right-click on a search result. Selecting that option will launch the web browser to the specific location in the document repository where the file is stored.

This seamless integration of Federated Search within Windows allows customers to greatly simplify their workflow for getting at remote files while still being able to easily take advantages of the advanced functionality of content repositories.

Simple to deploy

Our next challenge was to make it easy for customers to get these new connections onto their machines. It wouldn’t be practical to ship Windows with a connection to every solution in the world, so we shifted to a way that would make it very easy for any web service to deploy a connection to their specific service.

The model we came up with is similar to the way you add favorites from the web today. A web service can place a link to an .osdx file somewhere on their web page (see Channel 9’s search page for an example). The .osdx file is a simple XML file that uses the OpenSearch description document format to describe how to connect to the web service, and gives the web service some control of how the data is presented in Windows Explorer. When a person clicks on the link, Windows performs an ultra-lightweight install process that adds a search connector to that web service and places a link to that it in the Windows Explorer favorites.

If you are an administrator in an enterprise environment, you will likely want to provide some pre-installed search connectors for your users to search the company intranet or a popular internal SharePoint site for example. You can do this by deploying the search connector (.searchconnector-ms) files to your users’ machines via typical deployment techniques such as imaging, group policy preferences or startup scripts. The beauty is that it’s just a simple XML configuration file and there’s no code that needs to get installed on their machines. It’s also possible to pin one of these as a link from the Start menu through group policy. In the group policy editor look for the policy in this area: User Configuration> Administrative Templates > Windows Components > Windows Explorer. The policy name is “Pin Libraries or Search connectors to Search again links and start menu”.

Launching a Federated Search of an enterprise Intranet from the Start Menu

Launching a Federated Search of an enterprise Intranet from the Start Menu

Easy to adopt

Of course this technology depends on having services that support it. Although there are only a few services that provide a .osdx for you today, there are many existing services that already support the basic OpenSearch requirements.

We’re already seeing positive initial reactions from enthusiasts and ISVs alike echoing that it is indeed easy to enable your service to work with our Federated Search platform. If you’re a developer and want to enable an existing web based service to support Windows 7 Federated Search, you’ll need to provide a web service that accepts an http GET request with the search terms embedded somewhere in the URL and be able to return the results as an RSS or Atom feed. These requirements are typically very easy to meet for most applications that already provide search services via a web browser.

Your web service results should include the basic RSS tags like <link>, <title>, <description>, <pubDate> to get started but there’s much more that you can include in the RSS output and customization you can do within the .osdx file to enhance the experience for the end user.

For more information, we’ve published the Windows 7 Federated Search implementer’s guide with detailed information on how to enable your data source to work with Windows Federated Search. There’s also a recorded PDC session that demonstrates how to build a Windows Federated Search compatible web service for an existing SQL database.

- Brandon Paddock, Scott Dart & Alwin Vyhmeister, Find and Organize

Leave a Comment
  • Please add 5 and 5 and type the answer here:
  • Post
  • Hey guys, I just wanted to point out that a lot of these topics of discussion should be coming up as posts of their own pretty soon, and that those posts may help shed some light on the changes made in the Win7 Explorer.

    To address a couple of quick comments:

    @unifex - Have you used the Windows 7 Beta?  I ask because some of your comments don't apply.  For example:

    * Desktop does show in the nav pane by default, it's the first entry.

    * There is no sliding around in the folder tree, that was only in Vista.

    * Windows 7 will always remember the view settings you choose.  In Libraries, the view options are always the same regardless of what type of file you put in them, there is no "sniffing."  And in regular folders that should only happen the first time you browse to a folder.  A bunch of bugs were fixed in this area since Vista (and more since the Beta).

    I'm also curious about how you think the current Explorer design impedes your ability to organize your files effectively.  Do you not make use of Libraries at all?  They can be very helpful in organizing your files how you like.  However, if you don't use them, you can collapse that node in the nav pane and it will stay that way.  Same goes if you expand the Computer node.  Plus you can put whatever you like into Favorites.

    You can also use the settings dialog, or right-click on the background of the Navigation pane area to choose the "Show All Folders" option, which I think feels very much like a 'classic' Explorer model.

    @Sevenflavor -

    Where are you seeing a problem with the size column?

    File types like AVI and TXT do not provide support for open metadata.  This is a limitation of the file format.  Believe me, we aren't ignoring this problem, but unfortunately some problems don't have easy answers and take more time than others to get right.

    Fortunately, the newer container formats like WMV, MOV, MP4, etc do support open metadata and properties like Tags / Comments, etc.  And at least in Windows 7 they have property handlers built-in.

  • I meant showing folder size in columns, esp since IColumnProvider interface is gone. And yes, I've read Raymond Chen's explanation on it on how it would increase SMB chattiness, but you could disallow that across networks then.

  • Windows search doesn't add the subfolders to the index, while adding the whole drive to index location. I added every drive partition to the search location, but still folders in partitions other than Win7 system partition, are not getting indexed. I have to manually add every single folder to index lcoations.

    Also, are you guys anyhow interested in the feedback, that I submitted using feedback tool??

  • I think one of the primary problems most power users are having with the new Explorer model is the vast amount of virtual folders. AppData in your User Folder is a TERRIBLE place to allow programs to install components. This is not a "user file" in any way. AppData should be within "Program Files" where it belongs. Also, "ProgramData" is a similarly out of place folder. With that said, the other main problem with these two folders is that SOME programs seem to choose at RANDOM to install into one of them, or the sub-folders within, making it downright annoying to try and find these if something goes wrong (as was the case with an XP program I was using a few weeks ago). This simply makes no sense: any programs that wish to install in this fashion should just install in the root of AppData under their program files folder name, and then they can be easily found / edited.

    I, personally, have no problem with the User folder layout (except perhaps for folders which not all users use / need: contacts, searches, tracing..., as well as the misleadingly labeled "Links" which places shortcuts under the Favorites tab in the sidebar, which I find odd.) and think that, for the most part, it is a logical way of sorting data if you simply create sub-folders within each of these main groupings.

    I have posted my other thoughts above.

    I very much hope that mine (and everyone else 's) issues are addressed for the RC release, especially in regard to the virtual folders disaster (which was the case in vista as well).

    - AeonSlayer / Simon

  • @bpaddock - indeed, I use both Vista and Win7 beta as a dual boot, so maybe I confused some features of the two. You are right about sliding - that's a Vista feature, sorry about that. The Desktop, however, I might be blind or something, but it's just not there. My beta is build 7000, the one that was available officially, I am not using pirated OS, so maybe you now have a newer build where this is resolved. But with my build this is not only the problem with the folder pane. Say I want to open a file in some application. It will then open a special explorer window as it was always doing it. Now in Win7 there is no Desktop! I have to literally dig into the Users folder finding where the Desktop actually located and that's not nice.

    Now as far as the customization of folders. Basically what I want is to have most folders to appear in exactly the same way, except for a few special ones, like Desktop or Control Panel. However, every now and then I create a new folder and then if I put an image file there Windows immediately assigns some other template to this folder - like showing thumbnails where I don't want them.

    One of the reasons I want to have explorer to behave the way I want is that often I use my computer for scientific computations and as a result I have lots of files with "extensions" that are not recognized by Windows. Therefore any sort of sorting or templates based on "file type" is often useless to me.

    Finally, sometimes I install programs that I need for a short time period only. Then I uninstall them, but unfortunately - and I am not sure whether to blam the developers or Windows - the installer/uninstaller leaves behind lots of files. Not that this takes up a lot of space, but still, I like to delete them. This means that I need to see "hidden" or "system" files. In XP, I could choose to see these files in those folders where I need it, but not in others. In Vista and also in Win7 if I choose this in Folder Options, then immediately all folders will start showing the hidden files, and in particular I will have two .ini files on the Desktop. There simply does not seem to be a way to choose such folder options for different folders on individual basis as it was in XP.

    Now, you said that a number of bugs were fixed since beta. As I mentioned, I only have the build 7000, since that's what was made officially available. Hopefully the Explorer will behave better in the next build I will be able to get (I am sort of apprehensive to installing pirated OS copies - I am not sure I could trust those guys that much).

  • @unifex -

    The "Desktop" link should be the first entry in the Favorites node on a clean install, even in the beta release.  It should be there along with Downloads and Recent Places unless you deleted the link, or replaced the folder containing them.

    This should also show up inside the Common File Dialog giving you the same easy access to the Desktop folder or any other location you choose to pin there.

    I'm also a bit unclear about your "hidden files" comments.  I'm quite sure Windows XP had exactly the same options as Windows Vista and Windows 7 for hidng or showing hidden and system files.  There was never an option to only show hidden files in certain folders, it has always been a global Explorer setting.

    If you enable "show hidden files" that will not cause desktop.ini files to appear.  Those should only show up if you also disable the "Hide protected operating sytem files" option.

    Have you tried the "Show All Folders" option for the navigation pane that I mentioned?  I'm interested to know if you find that more to your liking.

  • @bpaddock -

    Well, you're right of course:

    the "Desktop" link is there in the Favorites. However, I should say that this was the first time I actually clicked on Favorites. Previously, Favorites was just a list of internet bookmarks and I never saw a reason for them to be present in the file manager (in fact, in Windows 2000 I have actually removed the Favorites link from Windows Explorer. Later however I decided it was not worth the effort and just let it be there in XP and Vista).

    Now, when I start Explorer, I get separately Favorites, Libraries, Computer, and Network nodes. I keep using the old trick of starting the Explorer in my designated home directory - not anywhere in Users folder though, in fact I keep my files and the OS on different partitions. Before the Desktop was always right there, at the top of the tree, so to speak.

    Aha, now I see what you mean. "Show All Folders" does the trick!

    Thanks for tip and for being patient with me!

    Finally, yes, the desktop.ini files show up when both "show hidden files" is enabled and "hide protected ... files" disabled. However, when I do that in XP, no desktop.ini files show up on the desktop.

  • Brandon, any comment on why we're not allowed to index folders on network-attached hard drives? You've done a great job of telling us what doesn't work on other forums, but I've never been able to find an explanation as to -why- we're not getting this functionality. It seems like everyone that requests it is ignored.

    I know this article mentions that in an enterprise setting you'd rather not duplicate an index of the same server on everyone's machine, but I don't see why the option isn't allowed. My hard drive isn't a server -- it doesn't have the capability to index itself. Some insight into the decision to not allow me to index it would be thoroughly appreciated.


  • From a UX perspective, there are still three fundamental problems with Windows Explorer that take up far too much user time:

    1. No tabs. For example, why should I have to snap two explorer windows to my fullscreen desktop just to copy files from one window to the other, when I could have ONE window open and drag files back and forth between tabs? The screen space issue is non-issue in this scenario. (Or if not tabs, dragging back and forth through thumbnails, which gives a tab-like experience while lowering the difficulty of drap-and-drop.)

    There's a reason firefox stormed ahead, and why every other browser now uses tabs. Tabs increase productivity. I need that gain in my explorer window (as should anyone who has 6-14 windows open, which is your 70% of Windows users according to your own data). Considering you already have tabbed thumbnail support in 7, I'm very surprised this feature was not included with the other UI changes.

    2. Resizing column size. Windows Vista rarely shows the correct column width for filenames, leaving many filenames cut off by an "..." and a seemingly wasted whitespace on the far end by default. This screenshot is typical (although they've replaced the whitespace with additional columns):

    The most important of a file explorer -should always- be the file names. When you open up a new explorer window, all filenames up to a reasonable length (the current is not reasonable) should be easily available. The rest is secondary information.

    3. Finally, more relevant: search does not display the full file path in a manner that can be instantly grasped. This was an issue for me when I first installed Vista. This is an issue with me today. I hate hate HATE not being able to see the full file path of files in search. The current layout is not intuitive as there is no obvious indication what the relationship is between the name, the file path in parenthesis, the actual file location, and the current folder. All this information must be inferred by the user based upon where they were before the search happened.

    This is at very least a lack of proper feedback, and at most a poor ordering of results, again missing where the emphases should be considering the window's function. (I consider it the latter, as technically you're showing two results in one column, something not done anywhere else in the UI--at least so far as I have seen--and that should violate at least one of your design principles right there).

  • This post is about a new feature based on Search that allows searching across PCs and even servers in an Enterprise setting.

  • These products typically provide great features for team collaboration, document versioning and workflow management, archiving, retention policy enforcement, and other centrally-managed functionality that IT managers appreciate.

  • Enterprise customers have asked us for a solution that simplifies finding important content in these various data stores but without leaving their normal Windows work flows.

  • This is largely thanks to the prevalence of OpenSearch and RSS enabled clients (including Internet Explorer and Microsoft Search Server, among many others).

  • This works for any existing application that supports the Common File Dialog, and there are a whole lot of them!

Page 3 of 4 (60 items) 1234