Engineering Windows 7

Welcome to our blog dedicated to the engineering of Microsoft Windows 7

Federating Windows Search with Enterprise Data Sources

Federating Windows Search with Enterprise Data Sources

  • Comments 60

The Windows Explorer has evolved by enabling you to find all sorts of content by searching for it. Many of you have used the search features in Windows Vista (based on our instrumented data) from the start menu or from the search box in Explorer. It has been a long time since most of us could remember where everything is by carefully managing our folder hierarchy and finding things based on file name alone.  We often rely on domain specific search (in music players, mail clients, photo clients) but with Windows Vista and Windows 7 we make it possible to search within a namespace and across namespaces.  This post is about a new feature based on Search that allows searching across PCs and even servers in an Enterprise setting.  Alwin and Scott, program managers, and Brandon, a developer, on the “Find and Organize” feature team authored this post. --Steven

Finding your stuff

Whether you’re searching or browsing, Windows Explorer is really about finding your stuff, and once you’ve found it, doing something with it (such as copying, opening, deleting, etc). For data that lives on your PC or home network, Windows 7 has invested in HomeGroup and Libraries (subjects for a future posting from our team) to provide an easier and richer experience than ever before. However, we didn’t stop there. Over the last few years, we’ve seen enterprise customers’ important content migrate towards (or aggregated in) centralized content stores, such as SharePoint. These products typically provide great features for team collaboration, document versioning and workflow management, archiving, retention policy enforcement, and other centrally-managed functionality that IT managers appreciate.

Where Enterprise Data Lives

Important enterprise data is found on local machines, in a variety of centralized content stores and also beyond the firewall

Unfortunately, this has placed an extra burden on customers to learn each new content store’s user interface, often asking them to give up familiar desktop features like drag-and-drop. Given their collaborative focus, these sites grow organically and it can become hard to remember where a particular document was stored and then wade through long lists of them every time you want to get back to it. Enterprise customers have asked us for a solution that simplifies finding important content in these various data stores but without leaving their normal Windows work flows.

As we looked at this trend and the lack of integration with content management and content indexing web services, we used these guiding principles in developing a solution:

  • Natural for people to use. Customers want a more consistent experience for finding and working with data in these disparate content stores, and would like us to bridge the local and remote content experiences by helping them “roll over” from one to the other.
  • Easy for IT admins to deploy. IT admins don’t like to deploy code, and want low-maintenance solutions that are easy to manage. Meanwhile customers want to connect up these sources without going through long and tedious installation processes or having to get help every time they want to set up a new search location.
  • Easy for developers to adopt. Developers want to enable this functionality in their offerings quickly and easily. There are a lot of data sources which need to be supported because IT folks don’t want to be locked in to a specific server technology.

Choosing to build Federated Search

Federated Search wasn’t the only way to address these challenges. The brute force approach would have been to take our existing Windows Search indexing technology and just use it on these content stores—that index the remote content on a local PC. This isn’t a very realistic solution since it’s inefficient to have all content indexed over the network by each person’s machine, especially when the content is changing at a rapid pace and represents a large corpus. Corporate retention policies may also prevent keeping even a local index of certain sensitive data.

Fortunately, there’s a better option – Federated Search. Federated Search enables you to search a remote web service from Windows explorer and get results back that you can act on like any normal file. The largest barrier to doing Federated Search has already been taken care of too. That is, most of these content stores are already indexed on the server, or at least on some server. There are several great offerings that will accomplish this, such as Microsoft Search Server. Not only do these servers index this content, but many of them already expose search results via a standard web protocol. This is largely thanks to the prevalence of OpenSearch and RSS enabled clients (including Internet Explorer and Microsoft Search Server, among many others).

For Windows 7, we’ve added support for Federated Search using OpenSearch v1.1 and worked to make the experience a seamless one. We found this solution strikes a good balance by leveraging the strengths of content services and the strengths of local file interactions within Windows.

Natural to use

Using Windows Explorer, people are familiar with several important user interface and interaction elements. They know how to use the navigation pane to change what they’re looking at. They know how to scroll around, how to select an item (or several), and they know how to double-click to open them. Most people know how to right-click for context-sensitive options related to their selection, or how to find those options presented in the command bar. They know they can drag and drop items to move them around. They know how to change view modes. We hope that they know how to search their current location using the search box, and in Windows 7 we think we’ve made it much easier to discover and use the Preview Pane to make sure they’ve got the right result.

Searching Sharepoint from Windows Explorer in Windows 7

Searching a SharePoint site using the new Federated Search support in Windows Explorer

Much of the usefulness of building Federated Search into Explorer is our ability to take advantage of this knowledge and familiarity. This may seem obvious once you see it in action, but behind the scenes there’s quite a lot going on to make all of this happen. For example, some applications such as Microsoft Word already know how to work with web URLs. So opening a Word document from a web server is fairly straightforward. But the majority of applications you’ll encounter really only understand how to open files on the local machine or via standard network file sharing protocols. This includes everything from the built-in software like Notepad and Paint, to third-party software like Photoshop or iTunes.

To handle this case, we implemented a “just in time” download solution, which will download the file to the internet cache before opening an application or taking actions (like using the SendTo menu) which require local files. This lets us offer searches that are very “lightweight” from a server load perspective, where we display metadata and icons or thumbnails without ever requesting the actual file. Then if you take an action like previewing or opening an item, we will do some behind-the-scenes work to make a local copy of the file only if necessary.

That enables us to work with the existing application ecosystem without asking anything of developers. However, applications can also take steps to offer even better functionality in many cases. For example, Windows Photo Viewer has added support for non-file items. So if you open a picture result in the built-in photo viewer, it’s the photo viewer that downloads the item, not Explorer. This may not seem like a big deal, but it lets the photo viewer enable the forward and back buttons to jump to the next or previous result – and it will download that image on-demand. Starting at the PDC we began reaching out to third-party ISVs to encourage them to implement similar enhancements for Federated Search scenarios, and we will continue to offer guidance on how to best integrate with all of the newest Explorer features.

Finally, we support all the standard clipboard and drag-and-drop operations. So if you drag a Word document from a Federated Search query onto your desktop, it will be copied there. You’ll even see the familiar Windows Explorer copy dialog, with progress indication, cancel ability, conflict resolution, and so on.

But wait, there’s more! Windows Explorer is a great tool that many customers know and love. But some people use it without even knowing it. Countless Windows applications make use of what we call the Common File Dialog. This is a special Explorer window that lets you find and choose items to be opened or inserted into your current application, without ever leaving it. If you’ve ever clicked File and then Open or Save in an application menu, you’ve probably seen some version of this dialog. PowerPoint, for example, uses the common file dialog to insert pictures. That means from inside PowerPoint you can click Insert Picture, select the Federated Search link for your image repository, search for the picture you want, and then insert it directly into PowerPoint. This works for any existing application that supports the Common File Dialog, and there are a whole lot of them!

Inserting a picture into PowerPoint’s using Federated Search

Inserting a picture into PowerPoint’s using Federated Search

Our Federated Search solution is all about simple lightweight access with a common, familiar user interface. This has a lot of benefits as we described above, but there are also cases where a server’s web interface will offer its own benefits. This might involve advanced query building, browsing, or server work-flow tasks, for example. So Windows 7 builds a bridge to these content repositories. After doing a search against a supported location, you will see a “Search on Website” button in the command bar which allows you to seamlessly send the query up to the service’s web interface in the default web browser. You’ll also see the “Open File Location” menu item when you right-click on a search result. Selecting that option will launch the web browser to the specific location in the document repository where the file is stored.

This seamless integration of Federated Search within Windows allows customers to greatly simplify their workflow for getting at remote files while still being able to easily take advantages of the advanced functionality of content repositories.

Simple to deploy

Our next challenge was to make it easy for customers to get these new connections onto their machines. It wouldn’t be practical to ship Windows with a connection to every solution in the world, so we shifted to a way that would make it very easy for any web service to deploy a connection to their specific service.

The model we came up with is similar to the way you add favorites from the web today. A web service can place a link to an .osdx file somewhere on their web page (see Channel 9’s search page for an example). The .osdx file is a simple XML file that uses the OpenSearch description document format to describe how to connect to the web service, and gives the web service some control of how the data is presented in Windows Explorer. When a person clicks on the link, Windows performs an ultra-lightweight install process that adds a search connector to that web service and places a link to that it in the Windows Explorer favorites.

If you are an administrator in an enterprise environment, you will likely want to provide some pre-installed search connectors for your users to search the company intranet or a popular internal SharePoint site for example. You can do this by deploying the search connector (.searchconnector-ms) files to your users’ machines via typical deployment techniques such as imaging, group policy preferences or startup scripts. The beauty is that it’s just a simple XML configuration file and there’s no code that needs to get installed on their machines. It’s also possible to pin one of these as a link from the Start menu through group policy. In the group policy editor look for the policy in this area: User Configuration> Administrative Templates > Windows Components > Windows Explorer. The policy name is “Pin Libraries or Search connectors to Search again links and start menu”.

Launching a Federated Search of an enterprise Intranet from the Start Menu

Launching a Federated Search of an enterprise Intranet from the Start Menu

Easy to adopt

Of course this technology depends on having services that support it. Although there are only a few services that provide a .osdx for you today, there are many existing services that already support the basic OpenSearch requirements.

We’re already seeing positive initial reactions from enthusiasts and ISVs alike echoing that it is indeed easy to enable your service to work with our Federated Search platform. If you’re a developer and want to enable an existing web based service to support Windows 7 Federated Search, you’ll need to provide a web service that accepts an http GET request with the search terms embedded somewhere in the URL and be able to return the results as an RSS or Atom feed. These requirements are typically very easy to meet for most applications that already provide search services via a web browser.

Your web service results should include the basic RSS tags like <link>, <title>, <description>, <pubDate> to get started but there’s much more that you can include in the RSS output and customization you can do within the .osdx file to enhance the experience for the end user.

For more information, we’ve published the Windows 7 Federated Search implementer’s guide with detailed information on how to enable your data source to work with Windows Federated Search. There’s also a recorded PDC session that demonstrates how to build a Windows Federated Search compatible web service for an existing SQL database.

- Brandon Paddock, Scott Dart & Alwin Vyhmeister, Find and Organize

Leave a Comment
  • Please add 3 and 4 and type the answer here:
  • Post
  • Re: Jalf's comments

    In my XP install I keep most of my files on a data drive using my own layout and ignore the default "My..." folders.  When 7 is released I'd like to embrace the folder 'system', but not if it remains a mess, and particularly not if it's locked to the OS drive.

    With XP I could tell Documents and Settings to reside on D:\ by using nLite.  I would like to be able to choose the physical location of my "Users" directory, preferably without "hacking" my own install disc.

    It's simply not an option to keep all my data on the OS drive, not to mention a bad practice if multiple drives are available.

    The other reason using default folders is somewhat 'hostile' is that while re-installing Windows (XP) it says it may erase the documents folders, or the entire Documents and Settings.  (May be fixed in Vista, not sure)

  • Is there a way to get the visual search suggestions from the Windows 7 start menu?

    I would also love to see a way to use some alias to pick which search provider to use.  I am imagining that I could type 'msdn DataContractAttribute' and get results from MSDN.

  • Maybe a little bit OT...but how can we bring in ideas to improve Windows user experience? Dont think that its great to make pictures out of ascii letters in the send feedback tool...is it ok when we send you an E-Mail? Or how?

  • This looks promising. I am thinking we can have <inset favourite search engine> Image Search (for example) right from the common dialogue box, if the provider writes the web service. That would be really cool, although I expect copyright laws would stop it happening!

    sabya_sadhu: Look for a program called Sandboxie. It does what you're asking for.

  • 1) I certainly don't agree that

    "It has been a long time since most of us could remember where everything is by carefully managing our folder hierarchy and finding things based on file name alone."

    In fact, I would probably not trust someone to operate a computer if they weren't capable of putting things onto it and finding them later.

    It's why I hate the Vista/Win 7 explorer with such a passion. The assumption above has resulted in a design choice whereby Explorer (once the shining beacon of file explorers "done right") is now an application that actively tries to prevent me doing what I need and want to do with my own data. The only time I've had a problem with finding data on my PC, in fact, is when using Win 7 and the OS has decided not only that *it knows best* where to store my downloaded files, but that *I shouldn't be allowed to access the folder they're in* - not even as an admin user. I've no idea how that's an improvement.

    2) "Windows Explorer is a great tool that many customers know and love."

    WAS. Now, it is a nanny which is attempting to tell me how, where, and when I should organise my files. This started with User Directories (good idea, messy implementation) and the My Documents/Pictures/Video system. Libraries, however much they are repainted, are nothing other than a slightly less messy implementation of the same idea.

    As others have pointed out above, creating "user" folders for data as part of the OS drive is NOT a functional ideal. It's messy, and has numerous problems (data loss if your OS needs to be re-installed being the most obvious). Not to mention it's very badly policed and the majority of time it's little more than a garbage dump for temp files, installers, and setup folders for the sort of developer who think that all apps should have a constantly running background service.

  • Re: bananaman

    "With XP I could tell Documents and Settings to reside on D:\ by using nLite.  I would like to be able to choose the physical location of my "Users" directory, preferably without "hacking" my own install disc.

    It's simply not an option to keep all my data on the OS drive, not to mention a bad practice if multiple drives are available."

    I do that as well for for years in XP, although manually not through nLite - once Windows fresh installed i go and move My Documents (right click, Properties, Move). You can do the same in W7, go into your profile folders, right click My Documents > Location > Move..., you have to do it manually for each folder though. In XP it used that when cutting and pasting to move Favorites around it actually updated the locations in registry as well, you could've done that with XP TweakUI as well...

    Would be nice indeed to allow to move the entire profile files/folders to a separate partition in one go...

  • @Hairs -

    Thanks for the feedback.  I'm a little confused about two points, though.

    First, you said:

    "the OS has decided not only that *it knows best* where to store my downloaded files, but that *I shouldn't be allowed to access the folder they're in* - not even as an admin user."

    I'm not quite sure what you're referring to there.  By default, IE will download files to the Downloads folder under your user profile.  There is a shortcut to it in the Favorites of your Explorer navigation pane by default.  This location requires no special privileges to access.  Could you better explain the problem you're seeing?

    Second, your comments about the Libraries feature indicate you haven't discovered their full potential.  Libraries are designed to let YOU decide where your files are stored.  The OS doesn't tell you where to put them, you just add the locations you use to the appropriate Library.  Then it provides an aggregated view of those locations, offers instant search and rich views over those files, and allows other applications like Windows Media Player / Media Center (and soon many others) to share those same library definitions, which are all decided by you.

    If you want to add a location to a Library, try going to it and then clicking on the link that says something like "Includes 2 locations" to view and change the included locations.

    Hope that helps!  As we hinted at in this post, a detailed post about Libraries is in the works :)

  • Thanks manicmarc for your reply,exactly that i was talking about.But i want microsoft to integrate that technology into windows.We can't rely upon third party softwares you know?and i do know that microsoft can better implement that kind of technology in their own product.

    Regards, Sabyasachi Sadhu.

  • Brandon, Steven, all you guys, nice work but I have to complain.

    I, like many others, have a NAS on my home network for local storage, which Windows will not allow me to index. I don't believe this is fair. I understand the point about keeping indices on different machines as being redundant, but what am I to do? I have all my important content on a network share, Windows won't let me index it, and my Libraries experience suffers.

    What am I supposed to do here? (Don't say buy a Windows Home Server)

  • I feel the pain that everyone else here is commenting on...

    I feel the single worst thing for me with Explorer (slightly OT as many others are...) is I have a dual boot of 7 and Vista set up on my main laptop. Now, accessing my work files from Vista partition results in read-only mode, which is a real pain when trying to edit one of these files! You have to write a new copy to 7 part., make the alterations, and then re-paste it in the Vista part. Simply giving the user an EASY WAY to REMOVE read-only from THEIR OWN FILES would be very nice indeed. There are often similar problems with media playback, and general copying of files where they are read only or access denied when on other Part.s (or even on your main Part. sometimes) even when you are the admin. This is a major problem, as it doesn't put "you in control" at all.

    Just my honest thoughts.

    - AeonSlayer / Simon

  • I'm amazed that there's a "Find and Organize” feature team! I presumed there was a "Lose and Disorganise" feature team! I presumed the complexity and confusion was to avoid lawsuits with Total Commander or something. Or just to get grannies excited with pretty buttons.

    We have had to put up with:

    User folders

    Public folders

    My Music

    My Pictures

    My Documents

    Documents

    etc.

    and now

    Libraries

    What a confusing mess!

    I agree with all the criticisms above. I have the same problem of "access denied" in Windows 7, even to something obvious like My Documents. And I tried fixing the permissions etc. Hopeless.

    I just want to put all my files in one place and not have to look at My pictures, Libraries etc. Ever. Again. Please?

    Aside from programs, the main things I use are Windows Explorer and the Taskbar, and they are being fiddled with by MS. The most used things should be the most customiseable and revertable.

    I wished the filing system was back to Windows 95 where everything was where you want it and there were no distractions and confusions and customisations stayed put.

    There should be an option at install to say where you want all your files to go. Alternatively, Windows Explorer and Search should have the Ribbon interface where you can click everything you want and it stays that way until you change it. You should be able to delete My Pictures, Libraries, etc. ...forever!

    I want to go back to XP from Vista, but I can't. But the filing is even worse in Windows 7. But I'm glad you got rid of the Documents and Settings folder when you moved from XP to Vista.

    And it seems that the Windows 7 team haven't noticed that Search doesn't work in some scenarios! I can put in a CD with files on it, type in the file name that I see on the screen, and Search reports not found. Sometimes you have to dig into 4 submenus to find advanced search to search non-indexed locations or to find text within files.

    Anyway, good luck.

  • As many have already said above, it is a pity that all these nice innovations come at the expence of removing useful, time proven features from Windows Explorer. As a result, one is being forced to use third-party Explorer replacements.

    It's fine to have libraries and super search technologies, but why do you think nobody wants to keep their files properly organized? It looks like you are getting the idea from Goodle mail - they won't even let you create your own folders in their web interface, saying that search is better.

    In reality, search is not better since if you do not remember exactly how to spell the item  you're searching for or simply have just a vague idea of what you want to find no search will ever find it. What you need to do in such event is to look through all relevant files and folders and that requires knowing where they are and a good tool to look at your files.

    The Windows Explorer becomes less and less suitable for such a task, which makes no sense at all. Now, the left (folder) pane does not even show Desktop. And all this sliding around in the folder trees is just annoying.

    Also, Windows now thinks it knows better how to present my folders. I think maybe for grandmothers it would be useful, or to people who use their computer mostly as a media player. But I really want ot be able to set folder view options for different folders the way I want it and keep these settings, not to have them changed automatically if a picture file is added. I want to have a default setting for newly created folders independent of whether I am putting there a document or a video clip.

    There is very simple way to settle things - to give an option of classic explorer as it is in XP. This way those who want to deal with their own file structure will be happy and those who are content with a mess on their hard drives will use search.    

  • Oh, and of course for people like me, who are using Windows at home on a machine which is not really networked and keeping their files on a local hard drive - although definitly not on the same drive as the OS - a lot of the network related improvements are not relevant. The ability to manage owr own files is.

    I'm not saying that you should not innovate in all directions. I'm just saying that you should not forget home users either - and there are people who do not want to keep their files on remote servers.

  • Wow. I haven't had many of the troubles that some of you are having. If I need a photo on my computer, I just go straight to the My Photos folder. When I use Windows Explorer, I can see everything in My Photos and every other folder without any effort. I seem to search and find everything just fine.

    I think some of you guys are still trying to use Windows 7 like if it was Vista, XP, 2000, 98, or any of the previous version. With anything thats brand new, there's a learning curve requires time and patience. You have to "unlearn" what you've learned with the last Windows version. Its really not that difficult. At times it can be frustrating, but sometimes a good cheat sheet helps. Unfortunately, there isn't one for Windows 7 yet. Eventually, there will be.

    I just find it amusing and some what disappointing that people are using the next version of Windows like the last one. Thats like trying to drive your new car, like you drove your last car. Each vehicle has something different, some new quirks, and handles things differently. You have to learn it and people are so stubborn to change. I always approach a new version of Windows with a clear and empty mind. When I transitioned to Vista, the first thing I said was "This is not XP. Use it and learn." From then on it was child's play.

    Just some thoughts I had, as I read a lot of the comments on here. Hopefully, you'll find something useful in all the comments here.

  • You know, I'm surprised at how Microsoft goes to great lengths to improve features like search federation which will be used mostly by enterprise users and yet it doesn't care about important Windows Explorer issues like showing size in columns, folder size, size on status bar, storing metadata for files like AVI, MPEG, TXT!!, disabling annoying autorefresh/autosort behavior when working with files. These are the primary reasons I hate Windows Explorer in Vista so strongly and MS has done nothing regarding these particular issues in Windows 7.

Page 2 of 4 (60 items) 1234