Engineering Windows 7

Welcome to our blog dedicated to the engineering of Microsoft Windows 7

Windows Desktop Search

Windows Desktop Search

One of the points of feedback has been about disabling services and optionally installing components—we’ve talked about our goals in this area in previous posts.  A key driver around wanting this type of control (but not the only driver) is a perception around performance and resource consumption of various platform components.  A goal of Windows is to provide a reliable and consistent platform for developers—one where they can count on system services as being available, as well as a set of OS features that all customers have the potential to benefit from.  At the same time we must do so in a way that is efficient in system resource usage—efficient enough so the benefit outweighs the cost.  We recognize that some percentage of customers believe solving this equation can only be done manually—much like some believe that the best car performance can only come from manual transmission.  For this post we’re going to look into the desktop search functionality from the perspective of the work we’re doing as both a broadly available platform component and to provide the rich end-user functionality, and also look at the engineering tradeoffs involved and techniques we use to build a great solution for everyone.  Chris McConnell, a principal SDE on the Find and Organize team, contributed this post.  --Steven

Are you one of those folks who believes that search indexing is the cause of your drive light flashing like mad? Do you believe this is the reason you’re getting skooled when playing first person shooters with friends? If so, this blog post is for you! The Find and Organize team owns the ‘Windows Search’ service, which we simply refer to as the ‘indexer’. A refrain that we hear from some Vista power-users is they want to disable the indexer because they believe it is eating up precious system resources on their PC, offering little in return. Per our telemetry data, at most about 1.5% of Vista users disable the indexing service, and we believe that this perception is one motivator for doing so.

The goal of this blog post is to clarify the role of the indexer and highlight some of the work that has been done to make sure the indexer uses system resources responsibly. Let’s start by talking about the function of the indexing service – what is it for? why should you leave it running?

Why Index?

Today’s PCs are filled with many rich types of files, such as documents, photos, music, videos, and so on. The number of files people have on their PC is growing at a rapid pace, making it harder and harder for them to find what they’re looking for, no matter how organized their files may (or may not) be. Increasingly, these files contain a good deal of structure, with metadata properties which describe their contents. A typical music file contains properties which describe the artist, album name, year of release, genre, duration of the song, and others which can be very useful when searching for music.

Although search indexing technologies date back to the early days of Windows, With Windows Vista Microsoft introduced a consumer operating system that brought this functionality to mainstream users more prominently. Prior to Vista, searching was pretty rudimentary – often a brute force crawl through the files on your machine, looking only at simple file properties such as file name, date modified, and size, or an application specific index of application specific data. Within Windows, a more comprehensive search option allowed you to also examine the contents of the files, but this wasn’t widely used. It was fairly basic functionality – it treated all files just the same, without the tapping in to the rich metadata properties available in the files.

In Windows Vista, the indexing service is on by default and includes expanded support in terms of the number of file formats and properties which are indexed. The indexer watches specific folders on your PC and catalogues their contents to facilitate fast searching of those files. When Windows indexes your music files, it also knows how to extract the music-specific properties which you’re most likely to search for. This enables support for more powerful searches and richer views over your files which wasn’t possible before. But this indexing doesn’t come free, and this is where engineering gets interesting. There’s a non-zero cost (in terms of system resources) that has to be paid to enable this functionality, and there are trade-offs involved in when and how you pay that price. There is nothing unique to indexing—all features have this cost-benefit tradeoff. 

Trade-Offs

Many search solutions follow(ed) the traditional “grep” model which means every search will read all of the files you wanted to search. In this case, you paid with your time as you waited for the search to execute. The more files you searched, the longer you waited each time you searched. If you wanted to perform the same search again, you would “pay” again. And the value you were getting in return wasn’t very good since the search functionality wasn’t particularly powerful. With Windows Vista , the indexer tries to read all of your files before you search so that when you search, it’s generally quicker and more responsive. This requires the indexer to scan all of your files just once initially, and not each and every time you perform a search. If the file were to change, the indexer would receive a notification (a “push” event) so that it could read that file again. When the indexer reads a file, it extracts the pertinent information about the file to enable more powerful searches and views. The challenge is to do this quickly enough so that the index is always up to date and ready for you to search, but also doing so in such a way that it doesn’t impact the performance of your system in a negative way. This is always a balancing act requiring trade-offs, and there are a number of things the indexer does to maintain its standing as a good Windows citizen while working to make sure that the index is always up-to-date.

A Model Citizen

A lot of work has gone into making the indexer be a model Windows citizen. We’ve written an extensive whitepaper on the issue, but it’s worth covering some of the highlights here. First and foremost, the indexer only monitors certain folders, which limits the amount of work it needs to do to just those files that you’re most likely to search. The indexer also “backs off” when you are actively using your PC. It indexes files more slowly, or stops entirely depending on the level of activity on the PC. When the indexer is reading files it uses low priority I/O and CPU and immediately releases the file if another application needs access.

It’s critical that we get all of these issues right for the indexer, because it’s not only important for the features that our team builds (like Windows Search), but it’s important to the Windows platform as a whole. There are a host of applications which require the ability to search file contents on the PC. Imagine if each one of those applications built their own version of the indexer! Even if all of these applications did a great job, there will be a lot of unnecessary and redundant activity happening on your PC. Every time you saved one of your documents there will be a flurry of activity as these different indexers rushed to read the new version. To combat that, the indexer is designed to do this work for any application which might choose to use it and provide an open platform and API with flexibility and extensibility for developers. The API designed to be flexible enough to meet needs across the Windows ecosystem. Out of the box, the indexer has knowledge of about 200 common file types, cataloging nearly 400 different properties by default. And there is support for applications to add new file types and properties at any time. Applications can also add support for indexing of data types that aren’t file-based at all, like your e-mail. Just a few of the applications that are leveraging the indexer today are Microsoft Office Outlook and OneNote, Lotus Notes, Windows Live Photo Gallery, Internet Explorer 8, and Google Desktop Search. As with all extensible systems, developers often find creative uses for components for the system services. One example of this is the way the Tablet PC components leverage the index contents to improve handwriting accuracy.

Constantly Improving

We’re constantly working to improve the indexer’s performance and reliability. Version 3 shipped in Windows Vista.  Major improvements in this version included:

  • The indexer runs as a system service vs. as a per user process.  This minimizes impact on multi-user scenarios e.g. only one catalog per system results in reduction in catalog size and prevents re-indexing of the same content over and over.  Additional benefit is gained from the robust nature of services.
  • The indexer employs low priority I/O to minimize impact of indexing on responsiveness of PC.  Before Windows Vista, all I/O was treated equally.

We’ve already released Windows Search version 4 as an enhancement to either Windows XP or Vista which goes even further in terms of performance and stability improvements, such as:

  • Significant improvements across the board for queries which involve sorting, filtering or grouping. Example improvements on Vista include:
    1. Getting all results while sorting or grouping has been improved. Typical query improvements  are up to 38% faster.
    2. CPU time has been reduced by 80%
    3. Memory usage has been reduced by 20%
  • Load on Exchange servers is reduced over 95% when Outlook is running in online mode.  With previous versions of Windows Search, large numbers of Outlook clients running in online mode could easily overwhelm the Exchange server.
  • Reliability improvements including:
    1. We made a number of fixes to address user-reported situations that previously caused indexing to stop working.
    2. We improved the indexer’s ability to both prevent and recover from index corruptions.  Now, when catalog corruption is detected it is always rebuilt automatically – previously this only happened in certain cases.
    3. We added new logging and events to help track down and fix reliability issues.

And we’ve done even more to improve performance and reliability for the indexer in Windows 7 which you’ll soon see at the PDC. If you still believe that the indexer is giving you trouble, we’ve got a few things for you to try:

  • Download and install Windows Search 4 (on Vista or XP).
  • Download and install the Indexer Gadget from the Windows Live Gadget Gallery (Vista only). This gadget was written by one of our team members, and gives you a quick way to view the number of items indexed. It also allows you to pause indexing, or to make it run full-speed (without backing off).
  • If you‘re one of those people who like to get under the hood of the car and poke around the engine, you can use the Windows Task manager and/or Resource Monitor to monitor the following processes: SearchIndexer, SearchFilterHost, SearchProtocolHost.

If you feel as though your system is slow, and you suspect the indexer is the culprit, watch the gadget as you work with your PC. Is the number of indexed items changing significantly when you’re experiencing problems? If you pause the indexer, does your system recover? We’re always looking to make our search experience better, so if you are still running into issues, we want to hear about them. Send your feedback to idx-help@microsoft.com.

Chris McConnell

Find and Organize

Leave a Comment
  • Please add 6 and 1 and type the answer here:
  • Post
  • I agree that Windows Desktop search is nice, but why does it need an icon in my taksbar? See the previous posts in this blog. (If there is an option to turn it off, I have missed it).

    Also, It needs a way to fall back to traditional searching. After installing it, I can't find a way to search my Windows and Program Files directories. This is necessary for some tasks, such as manual virus removal.

  • Thanks for the explanation about how "Search" works.

    My biggest concern is not related to performance slowdown as much as GUI and results of searches:

    If I choose to search "Everywhere" and check the option to include "Non indexed, hidden etc. etc." the search never ends; what I mean is that I have the progress bar hanging almost at the end...forever ( I tested up to 24 hours).

    Others have already posted about the issue about Vista and laptop batteries so I will not go there; what I wish to see back in WIndows 7 is "Virtual Folders". They were a great concept and I would like to see them back the same way they were implemented.

    Virtual Folders alone would be enough for me to move to Windows 7 without delays.

    Thanks again

    Fritzly

  • I don't doubt that WDS tries to efficiently balance the resources available for indexing files.  However, I have several problems with it:

    1) The UI is less powerful than what is available in Windows 2000, XP, 2003, particularly for searching non-indexed locations.  In XP it is very easy to right-click on a folder and use advanced search options to find a particular file.  This is often useful for people working in IT in particular, and for others as well.

    2) On my Vista system at home the hard drive seems to be constantly running, this is not the case with XP. I think that search is probably only one component of this, but it adds to the perception that Vista is a resource hog (even with low priority I/O, etc.).  

    3) Another UI comment, that is only peripherally related to search, is the feature where Vista tries to automatically determine how to display folder detail information.  In my experience this does not work well for how I want to store files.  I would like to always have details view with standard details like name, size, date modified.  However, Vista changes the selected details based what files are in a folder.  This does not work well for me because I often have folders with different types of files and the details that Vista chooses to display are not the ones I want.  The reason I'm mentioning this problem in response to this blog post, is because it is an example of system customization problems in Vista (which this post talked about).  

  • As a "l33t g4m3r" I find myself very pleased with Vista. I think you guys are doing a great job developing Windows 7.

  • ok

    Windows Search is Great but you Wont Eye candy ;)

    Feedback resolved :D

  • I've never had my system going slow due to search (to my knowledge). I very rarely use it for anything more than a filename search. Old habbits I guess.

  • However good windows search gets, and however well you manage to optimise it, the simple fact is that windows is doing it backwards.

    There should be no background indexing at all. No matter how much you minimise it, it's there. What *should* have happened is that a new file system should have been created where updating the index was part of the file create/modify API. That would mean you *always* had an up to date index, and the system would *never* need to be scanned.

    Of course, if Vista had a proper explorer, you wouldn't need to search in the first place....

  • @ lyesmith:

    Yes, Windows Search 4 is part of Windows Update. He put the download link for those with Windows Update disabled.

  • The biggest disappointment in Windows Vista for search has been its user interface. The Start menu search is nice, the indexed search integrated in Explorer also works well. What I don't like very much is the real-time non-indexed search which is slower than what used to be in XP, and the UI for non-indexed advanced search is very minimal and depends upon users to memorize the advanced search query syntax.

    Simple features like specifying where to search are too deep inside the UI, several others e.g. you cannot specify a date range from the UI, searching by file extension aren't exposed in the GUI. Users hate to remember the search syntax especially when it is so flexible and comprehensive. I just want a simple UI which exposes all the search features which isn't in Vista's search UI and this is what keeps me from using Advanced search and instead use third-party utilities, though the Start menu search and integrated search UI in Explorer is useful and awesome. XP also very conveniently allowed starting the Search Explorer pane by one click from the toolbar button which was unfortunately taken away in Windows Vista. Lesson to learn is make the search GUI very flexible, customizable, comprehensive and productive-not taking too many clicks and Browse boxes.

  • Since we are on the topic of search, what would be nice if if we can chose weather or not to have the search in the start menu or on the desktop it self, even on the taskbar would be nice..

  • One thing that I don't understand is that if I search for files in the start menu, it will show the most relevant match in the top, which is good; but if I press enter it pops up another window, which is not what I want: I want to it open the most relevant file. Just like it does when I search for programs.

  • nice...

    main task for operating system is running programs. we can read about everything on this blog, but not about the most important things.

    Steven, please speak about concretes... What will be changes in system to make this main function better (faster, more secure, etc.) ? Does your silence mean, that nothing ?

  • @marcinw

    PDC

    13 day and Windows 7 disclouse :D

    http://microsoftpdc.com/

  • marcinw, you aren't the only windows user in the world, other users care about these things, they will probably address what you care about in a future post, so wait till then to make your unclever slashdotian comments, please.

  • Call me old school but I know where I put stuff on my PC. Back from the DOS days, you had to know where your stuff was!

    It's a bit of a no brainer isn't it??? Make a new directory call it something sensible, next time you need it there it is...

    I think "Search" is great when I'm working on other peoples pooters. One of the most common things I hear is "I downloaded that file but I can't find it!" So yea, great feature give the numptys a comprehensive search tool. I wonder what these peoples offices are like, a pile of papers thrown in the general direction of the filling cabinet...

    Why not meet us half way if we can't get rid of it because other apps may depend on it. Then please let us have some control. You can put the controls in "Administrative Tools" if you like. Just let us schedule the indexing and hide the search box on the menu. Out of sight out of mind.

    To be honest most of these issues where power users want control but MS for the reasons of dependency need the item not to be disabled, could be dealt with by putting something in the "Administrative Tools".

    Save us having to fire up regedit and make our own nips and tucks. Because you *know* that we are going to do just that. The end result will be that some of us will be running a version of Windows that simply isn't what you intend.

Page 2 of 10 (139 items) 12345»