Engineering Windows 7

Welcome to our blog dedicated to the engineering of Microsoft Windows 7

Windows Desktop Search

Windows Desktop Search

One of the points of feedback has been about disabling services and optionally installing components—we’ve talked about our goals in this area in previous posts.  A key driver around wanting this type of control (but not the only driver) is a perception around performance and resource consumption of various platform components.  A goal of Windows is to provide a reliable and consistent platform for developers—one where they can count on system services as being available, as well as a set of OS features that all customers have the potential to benefit from.  At the same time we must do so in a way that is efficient in system resource usage—efficient enough so the benefit outweighs the cost.  We recognize that some percentage of customers believe solving this equation can only be done manually—much like some believe that the best car performance can only come from manual transmission.  For this post we’re going to look into the desktop search functionality from the perspective of the work we’re doing as both a broadly available platform component and to provide the rich end-user functionality, and also look at the engineering tradeoffs involved and techniques we use to build a great solution for everyone.  Chris McConnell, a principal SDE on the Find and Organize team, contributed this post.  --Steven

Are you one of those folks who believes that search indexing is the cause of your drive light flashing like mad? Do you believe this is the reason you’re getting skooled when playing first person shooters with friends? If so, this blog post is for you! The Find and Organize team owns the ‘Windows Search’ service, which we simply refer to as the ‘indexer’. A refrain that we hear from some Vista power-users is they want to disable the indexer because they believe it is eating up precious system resources on their PC, offering little in return. Per our telemetry data, at most about 1.5% of Vista users disable the indexing service, and we believe that this perception is one motivator for doing so.

The goal of this blog post is to clarify the role of the indexer and highlight some of the work that has been done to make sure the indexer uses system resources responsibly. Let’s start by talking about the function of the indexing service – what is it for? why should you leave it running?

Why Index?

Today’s PCs are filled with many rich types of files, such as documents, photos, music, videos, and so on. The number of files people have on their PC is growing at a rapid pace, making it harder and harder for them to find what they’re looking for, no matter how organized their files may (or may not) be. Increasingly, these files contain a good deal of structure, with metadata properties which describe their contents. A typical music file contains properties which describe the artist, album name, year of release, genre, duration of the song, and others which can be very useful when searching for music.

Although search indexing technologies date back to the early days of Windows, With Windows Vista Microsoft introduced a consumer operating system that brought this functionality to mainstream users more prominently. Prior to Vista, searching was pretty rudimentary – often a brute force crawl through the files on your machine, looking only at simple file properties such as file name, date modified, and size, or an application specific index of application specific data. Within Windows, a more comprehensive search option allowed you to also examine the contents of the files, but this wasn’t widely used. It was fairly basic functionality – it treated all files just the same, without the tapping in to the rich metadata properties available in the files.

In Windows Vista, the indexing service is on by default and includes expanded support in terms of the number of file formats and properties which are indexed. The indexer watches specific folders on your PC and catalogues their contents to facilitate fast searching of those files. When Windows indexes your music files, it also knows how to extract the music-specific properties which you’re most likely to search for. This enables support for more powerful searches and richer views over your files which wasn’t possible before. But this indexing doesn’t come free, and this is where engineering gets interesting. There’s a non-zero cost (in terms of system resources) that has to be paid to enable this functionality, and there are trade-offs involved in when and how you pay that price. There is nothing unique to indexing—all features have this cost-benefit tradeoff. 

Trade-Offs

Many search solutions follow(ed) the traditional “grep” model which means every search will read all of the files you wanted to search. In this case, you paid with your time as you waited for the search to execute. The more files you searched, the longer you waited each time you searched. If you wanted to perform the same search again, you would “pay” again. And the value you were getting in return wasn’t very good since the search functionality wasn’t particularly powerful. With Windows Vista , the indexer tries to read all of your files before you search so that when you search, it’s generally quicker and more responsive. This requires the indexer to scan all of your files just once initially, and not each and every time you perform a search. If the file were to change, the indexer would receive a notification (a “push” event) so that it could read that file again. When the indexer reads a file, it extracts the pertinent information about the file to enable more powerful searches and views. The challenge is to do this quickly enough so that the index is always up to date and ready for you to search, but also doing so in such a way that it doesn’t impact the performance of your system in a negative way. This is always a balancing act requiring trade-offs, and there are a number of things the indexer does to maintain its standing as a good Windows citizen while working to make sure that the index is always up-to-date.

A Model Citizen

A lot of work has gone into making the indexer be a model Windows citizen. We’ve written an extensive whitepaper on the issue, but it’s worth covering some of the highlights here. First and foremost, the indexer only monitors certain folders, which limits the amount of work it needs to do to just those files that you’re most likely to search. The indexer also “backs off” when you are actively using your PC. It indexes files more slowly, or stops entirely depending on the level of activity on the PC. When the indexer is reading files it uses low priority I/O and CPU and immediately releases the file if another application needs access.

It’s critical that we get all of these issues right for the indexer, because it’s not only important for the features that our team builds (like Windows Search), but it’s important to the Windows platform as a whole. There are a host of applications which require the ability to search file contents on the PC. Imagine if each one of those applications built their own version of the indexer! Even if all of these applications did a great job, there will be a lot of unnecessary and redundant activity happening on your PC. Every time you saved one of your documents there will be a flurry of activity as these different indexers rushed to read the new version. To combat that, the indexer is designed to do this work for any application which might choose to use it and provide an open platform and API with flexibility and extensibility for developers. The API designed to be flexible enough to meet needs across the Windows ecosystem. Out of the box, the indexer has knowledge of about 200 common file types, cataloging nearly 400 different properties by default. And there is support for applications to add new file types and properties at any time. Applications can also add support for indexing of data types that aren’t file-based at all, like your e-mail. Just a few of the applications that are leveraging the indexer today are Microsoft Office Outlook and OneNote, Lotus Notes, Windows Live Photo Gallery, Internet Explorer 8, and Google Desktop Search. As with all extensible systems, developers often find creative uses for components for the system services. One example of this is the way the Tablet PC components leverage the index contents to improve handwriting accuracy.

Constantly Improving

We’re constantly working to improve the indexer’s performance and reliability. Version 3 shipped in Windows Vista.  Major improvements in this version included:

  • The indexer runs as a system service vs. as a per user process.  This minimizes impact on multi-user scenarios e.g. only one catalog per system results in reduction in catalog size and prevents re-indexing of the same content over and over.  Additional benefit is gained from the robust nature of services.
  • The indexer employs low priority I/O to minimize impact of indexing on responsiveness of PC.  Before Windows Vista, all I/O was treated equally.

We’ve already released Windows Search version 4 as an enhancement to either Windows XP or Vista which goes even further in terms of performance and stability improvements, such as:

  • Significant improvements across the board for queries which involve sorting, filtering or grouping. Example improvements on Vista include:
    1. Getting all results while sorting or grouping has been improved. Typical query improvements  are up to 38% faster.
    2. CPU time has been reduced by 80%
    3. Memory usage has been reduced by 20%
  • Load on Exchange servers is reduced over 95% when Outlook is running in online mode.  With previous versions of Windows Search, large numbers of Outlook clients running in online mode could easily overwhelm the Exchange server.
  • Reliability improvements including:
    1. We made a number of fixes to address user-reported situations that previously caused indexing to stop working.
    2. We improved the indexer’s ability to both prevent and recover from index corruptions.  Now, when catalog corruption is detected it is always rebuilt automatically – previously this only happened in certain cases.
    3. We added new logging and events to help track down and fix reliability issues.

And we’ve done even more to improve performance and reliability for the indexer in Windows 7 which you’ll soon see at the PDC. If you still believe that the indexer is giving you trouble, we’ve got a few things for you to try:

  • Download and install Windows Search 4 (on Vista or XP).
  • Download and install the Indexer Gadget from the Windows Live Gadget Gallery (Vista only). This gadget was written by one of our team members, and gives you a quick way to view the number of items indexed. It also allows you to pause indexing, or to make it run full-speed (without backing off).
  • If you‘re one of those people who like to get under the hood of the car and poke around the engine, you can use the Windows Task manager and/or Resource Monitor to monitor the following processes: SearchIndexer, SearchFilterHost, SearchProtocolHost.

If you feel as though your system is slow, and you suspect the indexer is the culprit, watch the gadget as you work with your PC. Is the number of indexed items changing significantly when you’re experiencing problems? If you pause the indexer, does your system recover? We’re always looking to make our search experience better, so if you are still running into issues, we want to hear about them. Send your feedback to idx-help@microsoft.com.

Chris McConnell

Find and Organize

Leave a Comment
  • Please add 5 and 8 and type the answer here:
  • Post
  • Isn't Windows Search 4 part of Windows Update? If not why not? Especially when WS3 part of Vista

  • I love Windows Search 4  on Vista!

    Research of any document is always accurate.

  • I love Windows Search in Vista at home.  Is there anyway I can replace XP's default search with Windows Search 4?  I use XP at work, and searching for files in painfully slow.  

    I can't stand the searchbox in the taskbar and would like to have it appear more integrated with XP.

  • Desktop search is great, however I still think more could be done. Would be even better if I could choose the indexing level. For example, I want the contents of 'Documents' and 'Music' to be indexed in depth; however I often just want to find a file to which I have the name in a non-indexed location such as 'Program Files'. It's crazy that I can search the text of a word document quicker than I can find an executable in a bin folder!

  • Don't bother making l33t g4m0rs happy. They often "know" inaccurate facts and will happily create useless registry keys because they think they'll make Windows go faster. They would turn off Search Indexer even if wouldn't have a measurable impact.

    Joe Average is slowly realizing that a blinking HDD LED doesn't mean a slow system - thanks to the I/O scheduler in Vista.

    And Windows Search 4 doesn't slow XP anymore like its predecessor. Even when indexing my +2GB PST-file... :)

    But Vista itself is not very I/O efficient. Instead of improving the indexer (which is good enough already) try to optimize 7 to need less and shorter I/O requests. Improve read-ahead (Anticipatory scheduling, anyone?). Reduce disk footprint of 7 by orders of magnitude. Resuming from hibernate can take ages on machines with lots of RAM.

  • Great blog explaining the pros and cons of the Windows Search.

  • Windows Search is nice and fast. What i missing is an advanced search UI. Where you can specify date, size, text in document, text in filename, extension, < > = relations, compare function etc.

    Id it is exist in vista than you need to make it more accessible.

    More responsive file system, ability to rename file while open aliases etc would be nice.

    But again. These are Vista features. What is new in Win 7. what will be the edge? Why should people switch to Win 7 from Vista or XP?

    With all these blog entries I can not see what will make the Win 7 a unique experience. What we are talking about here are design tweaks not fundamental changes.

  • I think the trouble with the perceived performance hit with the indexer isn't the indexer's fault at all. It's that while the indexer is well behaved, years and years of "background" scan services such as antivirus apps, or 3rd party search indexers have set a bad reputation towards anything that's perceived to do intensive work in the background.

    Vista's indexer is by far the most well behaved background scanner I've seen. Contrast that with, for example, Nero Scout, which I've had to disable on every single machine that came with Nero preinstalled, ever. Or any antivirus software in existence. In general, anything that tries to do anything else while I'm already doing something will be perceived as Bad (tm).

    Even in Vista. I remember, back before SP1 came out, one of my Vista machines would occasionally just sit there for hours, grinding away at the HD for no apparent reason. It wasn't using any CPU, and the culprit was some obscure service running from svchost, making it impossible to figure out from the task manager exactly what was it doing.

    It didn't cause any substantial slowdown, but just the fact it was doing something, and I couldn't even tell what it was, really ticked me off. And just because it was grinding away, everything "felt" slower because I'm mentally trained to associate "hard drive grinding noise" with "load/wait sequence".

    To this day, I still don't know what it was doing.

    I am curious as to how Windows determines it's a good time to scan. It wouldn't be just mouse and keyboard input... Maybe it would check if any DirectX apps are running? Background services? File IO? But then how would it detect other background scanners?

  • How does the indexer affect power usage on laptops? Even with power settings tuned for maximum power savings the same laptop with a fresh install of Windows Vista was getting 30-40% less time out of fully charged battery compared to Windows XP with the same set of programs. Another factor that contributes to the increased power usage in Vista is the graphics, but hard drive also drains considerable amount of power, especially in in seek mode.

    I am also curious to see statistics of how frequently people use search. Based on my experience, some people use it maybe once a month, and some don't even know that such functionality is available. Does it worth having the indexer working all the time? Does the speed of search out weight the increased power consumption over the lifetime of a computer?

    Would it make sense to enable the indexer only for specific programs, or when a specific type of search used for the first time?

  • PS.

    Many many thank's to Brandon P. for Gadget index ;)

  • It was briefly mentioned in an earlier comment, but levels of indexing thoroughness would be usefull. High detail indexing in Documents, and low detail indexing in Program Files, or even past a certain folder depth maybe?

  • Windows Search is a good approach, very useful, but Finder has more good looking, maybe a mix, like Windows Search for fast search and some like finder for advanced search would be great...

  • No need to index Program Files, since it indexes the Start Menu folder. Put a link in there and you are good. Who would want it to come up with all the support files??

  • Hard drives have default cluster size 4kb what means files get fragmented very easily into many millions of pieces. No wonder people get slow results with xp ,Vista has defragmenter integrated into weekly schedule whats a good things but it's intelligence and lack of boottime system file defrag and optimize placements won't give much horse power back into HDD.

    Programs could be smaller and with new standards , boot time system file optimizer should be scheduled with automatic updates with junk cleaner , visuals should have minimal impact and be wiser when PC gets low on ram or process power... Tons of stuff that should be invented and made into perfection...

    Glad that since Vista there was new rules for IO management, MFT defragment enabled , system restore from last session, new program prefetch but 5gigs and slow on older systems was big drawback and only hope is do wait how things finally will be given for new demanding users.

  • Whenever i build a PC with Windows the two things i shutoff first is the Indexing service and system restore. Fact is i rarely if ever search for files on my home PC at work mostly i search for mail items and for that i don't mind the short wait. Be nice if windows had a feature like locate on Linux its fast and effective after the initial scan.

Page 1 of 10 (139 items) 12345»