Engineering Windows 7

Welcome to our blog dedicated to the engineering of Microsoft Windows 7

Windows Desktop Search

Windows Desktop Search

One of the points of feedback has been about disabling services and optionally installing components—we’ve talked about our goals in this area in previous posts.  A key driver around wanting this type of control (but not the only driver) is a perception around performance and resource consumption of various platform components.  A goal of Windows is to provide a reliable and consistent platform for developers—one where they can count on system services as being available, as well as a set of OS features that all customers have the potential to benefit from.  At the same time we must do so in a way that is efficient in system resource usage—efficient enough so the benefit outweighs the cost.  We recognize that some percentage of customers believe solving this equation can only be done manually—much like some believe that the best car performance can only come from manual transmission.  For this post we’re going to look into the desktop search functionality from the perspective of the work we’re doing as both a broadly available platform component and to provide the rich end-user functionality, and also look at the engineering tradeoffs involved and techniques we use to build a great solution for everyone.  Chris McConnell, a principal SDE on the Find and Organize team, contributed this post.  --Steven

Are you one of those folks who believes that search indexing is the cause of your drive light flashing like mad? Do you believe this is the reason you’re getting skooled when playing first person shooters with friends? If so, this blog post is for you! The Find and Organize team owns the ‘Windows Search’ service, which we simply refer to as the ‘indexer’. A refrain that we hear from some Vista power-users is they want to disable the indexer because they believe it is eating up precious system resources on their PC, offering little in return. Per our telemetry data, at most about 1.5% of Vista users disable the indexing service, and we believe that this perception is one motivator for doing so.

The goal of this blog post is to clarify the role of the indexer and highlight some of the work that has been done to make sure the indexer uses system resources responsibly. Let’s start by talking about the function of the indexing service – what is it for? why should you leave it running?

Why Index?

Today’s PCs are filled with many rich types of files, such as documents, photos, music, videos, and so on. The number of files people have on their PC is growing at a rapid pace, making it harder and harder for them to find what they’re looking for, no matter how organized their files may (or may not) be. Increasingly, these files contain a good deal of structure, with metadata properties which describe their contents. A typical music file contains properties which describe the artist, album name, year of release, genre, duration of the song, and others which can be very useful when searching for music.

Although search indexing technologies date back to the early days of Windows, With Windows Vista Microsoft introduced a consumer operating system that brought this functionality to mainstream users more prominently. Prior to Vista, searching was pretty rudimentary – often a brute force crawl through the files on your machine, looking only at simple file properties such as file name, date modified, and size, or an application specific index of application specific data. Within Windows, a more comprehensive search option allowed you to also examine the contents of the files, but this wasn’t widely used. It was fairly basic functionality – it treated all files just the same, without the tapping in to the rich metadata properties available in the files.

In Windows Vista, the indexing service is on by default and includes expanded support in terms of the number of file formats and properties which are indexed. The indexer watches specific folders on your PC and catalogues their contents to facilitate fast searching of those files. When Windows indexes your music files, it also knows how to extract the music-specific properties which you’re most likely to search for. This enables support for more powerful searches and richer views over your files which wasn’t possible before. But this indexing doesn’t come free, and this is where engineering gets interesting. There’s a non-zero cost (in terms of system resources) that has to be paid to enable this functionality, and there are trade-offs involved in when and how you pay that price. There is nothing unique to indexing—all features have this cost-benefit tradeoff. 

Trade-Offs

Many search solutions follow(ed) the traditional “grep” model which means every search will read all of the files you wanted to search. In this case, you paid with your time as you waited for the search to execute. The more files you searched, the longer you waited each time you searched. If you wanted to perform the same search again, you would “pay” again. And the value you were getting in return wasn’t very good since the search functionality wasn’t particularly powerful. With Windows Vista , the indexer tries to read all of your files before you search so that when you search, it’s generally quicker and more responsive. This requires the indexer to scan all of your files just once initially, and not each and every time you perform a search. If the file were to change, the indexer would receive a notification (a “push” event) so that it could read that file again. When the indexer reads a file, it extracts the pertinent information about the file to enable more powerful searches and views. The challenge is to do this quickly enough so that the index is always up to date and ready for you to search, but also doing so in such a way that it doesn’t impact the performance of your system in a negative way. This is always a balancing act requiring trade-offs, and there are a number of things the indexer does to maintain its standing as a good Windows citizen while working to make sure that the index is always up-to-date.

A Model Citizen

A lot of work has gone into making the indexer be a model Windows citizen. We’ve written an extensive whitepaper on the issue, but it’s worth covering some of the highlights here. First and foremost, the indexer only monitors certain folders, which limits the amount of work it needs to do to just those files that you’re most likely to search. The indexer also “backs off” when you are actively using your PC. It indexes files more slowly, or stops entirely depending on the level of activity on the PC. When the indexer is reading files it uses low priority I/O and CPU and immediately releases the file if another application needs access.

It’s critical that we get all of these issues right for the indexer, because it’s not only important for the features that our team builds (like Windows Search), but it’s important to the Windows platform as a whole. There are a host of applications which require the ability to search file contents on the PC. Imagine if each one of those applications built their own version of the indexer! Even if all of these applications did a great job, there will be a lot of unnecessary and redundant activity happening on your PC. Every time you saved one of your documents there will be a flurry of activity as these different indexers rushed to read the new version. To combat that, the indexer is designed to do this work for any application which might choose to use it and provide an open platform and API with flexibility and extensibility for developers. The API designed to be flexible enough to meet needs across the Windows ecosystem. Out of the box, the indexer has knowledge of about 200 common file types, cataloging nearly 400 different properties by default. And there is support for applications to add new file types and properties at any time. Applications can also add support for indexing of data types that aren’t file-based at all, like your e-mail. Just a few of the applications that are leveraging the indexer today are Microsoft Office Outlook and OneNote, Lotus Notes, Windows Live Photo Gallery, Internet Explorer 8, and Google Desktop Search. As with all extensible systems, developers often find creative uses for components for the system services. One example of this is the way the Tablet PC components leverage the index contents to improve handwriting accuracy.

Constantly Improving

We’re constantly working to improve the indexer’s performance and reliability. Version 3 shipped in Windows Vista.  Major improvements in this version included:

  • The indexer runs as a system service vs. as a per user process.  This minimizes impact on multi-user scenarios e.g. only one catalog per system results in reduction in catalog size and prevents re-indexing of the same content over and over.  Additional benefit is gained from the robust nature of services.
  • The indexer employs low priority I/O to minimize impact of indexing on responsiveness of PC.  Before Windows Vista, all I/O was treated equally.

We’ve already released Windows Search version 4 as an enhancement to either Windows XP or Vista which goes even further in terms of performance and stability improvements, such as:

  • Significant improvements across the board for queries which involve sorting, filtering or grouping. Example improvements on Vista include:
    1. Getting all results while sorting or grouping has been improved. Typical query improvements  are up to 38% faster.
    2. CPU time has been reduced by 80%
    3. Memory usage has been reduced by 20%
  • Load on Exchange servers is reduced over 95% when Outlook is running in online mode.  With previous versions of Windows Search, large numbers of Outlook clients running in online mode could easily overwhelm the Exchange server.
  • Reliability improvements including:
    1. We made a number of fixes to address user-reported situations that previously caused indexing to stop working.
    2. We improved the indexer’s ability to both prevent and recover from index corruptions.  Now, when catalog corruption is detected it is always rebuilt automatically – previously this only happened in certain cases.
    3. We added new logging and events to help track down and fix reliability issues.

And we’ve done even more to improve performance and reliability for the indexer in Windows 7 which you’ll soon see at the PDC. If you still believe that the indexer is giving you trouble, we’ve got a few things for you to try:

  • Download and install Windows Search 4 (on Vista or XP).
  • Download and install the Indexer Gadget from the Windows Live Gadget Gallery (Vista only). This gadget was written by one of our team members, and gives you a quick way to view the number of items indexed. It also allows you to pause indexing, or to make it run full-speed (without backing off).
  • If you‘re one of those people who like to get under the hood of the car and poke around the engine, you can use the Windows Task manager and/or Resource Monitor to monitor the following processes: SearchIndexer, SearchFilterHost, SearchProtocolHost.

If you feel as though your system is slow, and you suspect the indexer is the culprit, watch the gadget as you work with your PC. Is the number of indexed items changing significantly when you’re experiencing problems? If you pause the indexer, does your system recover? We’re always looking to make our search experience better, so if you are still running into issues, we want to hear about them. Send your feedback to idx-help@microsoft.com.

Chris McConnell

Find and Organize

Leave a Comment
  • Please add 3 and 1 and type the answer here:
  • Post
  • I like Windows 7 desktop search.  Unfortunately, while I have Windows 7 installed on my home machine, search capability has disappeared.  I do not know whether I have done something inappropriate or not.  But I would like to re-enable search.  Any suggestions?

  • Hello,

    The internet is PACKED FULL of frustrated users who love Win7 but absolutely hate the removal of a friendly user interface for doing an advanced search.

    I have been a windows user for 20 years and still cannot work the Win7 search, having to remember commands like name: to be able to search for a name. It was FAR better when there was an INTERFACE to CHOOSE what you wanted rather than have to remember all sorts of codes and try get them right.

    The person/team who came up with this awful Win7 search should be replaced with a team who have a clue about what regular users would like in a search tool.

    Win7 took many steps forwards, but he search was a giant one backwards. I am searching for an alternative, sadly, because it is more convenient to be able to search from Windows Explorer rather than then open a 3rd party app.

    Regards,

    Mark Palmos,

    UK

  • I'm really happy that MS is taking care about plain users, but how about professionals? Why I need to go and use 3rd party grep in Win7/Vista or bust Search Assistant in XP when Win2000 had perfectly working search? If you want to make search "faster" by omitting 70% of the content from the search - be my guest, but leave working part of the tool alone!

  • If I type in a string in the searsh bar I should get back every file name and subfolder name with that string in it! It dont matter whether u index it or not, instead I get "NO FREQUING MATCHES FOUND" when im looking right at them on the same page or know they are in a subfolder. WHAT ARE U PPLE SMOKING? pull ur heads out of ur butts; its been decades and ur getting noticeably worse every minute!!

  • You are not constantly improving. The few seconds required to find those files I needed to locate with specific content in  XP and even Vista, which was not an improvement at all, were more than compensated by the ability to find content. Despite checking the option to search files and content in search options, search does not find those files with the needed information.

    Please add both the simple XP and the bolean based vista options to the next service pack or make them available.

  • I can't find a Download Link for Search-4 for Windows-7 !!!

    Please help.

  • Windows Search on - Task takes 15 minutes to complete. When OFF - same task takes 3 minutes.

    Measurements are consistent.

    Windows Search is extremely resource consuming and totally unfit for work environments.

  • Like Windows 7 - I think if Windows search is set up properly it should be a help rather than a hindrance!  www.emailsecuritymatters.com/.../windows-search-windows-7-outlook-2007

  • Would be nice to allow the old fashion type of search if needed.  

  • The fact that there is no way to search a directory in Windows 7 for files containing a particular string is a serious omission it used to be possible in XP.  Why do Microsoft have to keep making all their search products progressively worse!

  • Doesn't really answer the question. Do you need both? Indexing Service and Windows Search: yes or no. Thanks.

  • Our office previously used Windows XP with Windows Search 4.0 add-in.  Indiviual desktop users were able to index various folders and subfolders on various network drives.  We were also able to "snooze" the search function during a busy day (since it slowed the computer significantly) and restart it after business hours so that we always had a reasonably upto date index for fast searches and recall from the 1,000's of files.  There was a pulsating magifying glass on the task tool bar that let you know when you were indexing.  We have recently migrated to Windows 7 and I cannot discover how to continually index so that I have an up-to-date index from which to search.  Also, and more importantly, I cannot add network drives for indexing, even though I have access to the network drives.  I tried to download and install Search 4.0, but got a message that: "The version of Windows you have installed does not match the update you are trying to install."  Most of our IT people have never used Windows Search 4.0 add-in and don't appreciate the absence of this functionality in Windows 7.  Does anyone have any suggetstion of how to get the same functionality in Windows 7?

  • One of the biggest tradeoffs with search in Windows 7 is that you will never be able to find anything, even if the file you're looking for is staring you in the face.  

  • I have windows 7 OS on my laptop. When I need to search for a file containing a string of words I am not able to find it. In Windows XP it was not a problem at all. Please help.

  • I want to be able to complete my work and stay organized.  Why are you making it difficult to impossible?  I know my files are there!  How is it useful to have them and not be able to access them?

Page 9 of 10 (138 items) «678910