Notes on comments.
Welcome to our blog dedicated to the engineering of Microsoft Windows 7
One of the features that you’ve been pretty clear about (I’ve received over 100 emails on this topic!) is the desire to improve the disk defrag utility in Windows 7. We did. And from blogs we saw a few of you noticed, which is great. This is not as straight forward as it may appear. We know there’s a lot of history in defrag and how “back in the day” it was a very significant performance issue and also a big mystery to most people. So many folks came to know that if your machine is slow you had to go through the top-secret defrag process. In Windows Vista we decided to just put the process on autopilot with the intent that you’d never have to worry about it. In practice this turns out to be true, at least to the limits of automatically running a process (that is if you turn your machine off every night then it will never run). We received a lot of feedback from knowledgeable folks wanting more information on defrag status, especially during execution, as well as more flexibility in terms of the overall management of the process. This post will detail the changes we made based on that feedback. In reading the mail and comments we received, we also thought it would be valuable to go into a little bit more detail about the process, the perceptions and reality of performance gains, as well as the specific improvements. This post is by Rajeev Nagar and Matt Garson, both are Program Managers on our File System feature team. --Steven
In this blog, we focus on disk defragmentation in Windows 7. Before we discuss the changes introduced in Windows 7, let’s chat a bit about what fragmentation is, and its applicability.
Within the storage and memory hierarchy comprising the hardware pipeline between the hard disk and CPU, hard disks are relatively slower and have relatively higher latency. Read/write times from and to a hard disk are measured in milliseconds (typically, 2-5 ms) – which sounds quite fast until compared to a 2GHz CPU that can compute data in less than 10 nanoseconds (on average), once the data is in the L1 memory cache of the processor.
This performance gap has only been increasing over the past 2 decades – the figures below are noteworthy.
In short, the figures illustrate that while disk capacities are increasing, their ability to transfer data or write new data is not increasing at an equivalent rate – so disks contain more data that takes longer to read or write. Consequently, fast CPUs are relatively idle, waiting for data to do work on.
Significant research in Computer Science has focused on improving overall system I/O performance, which has lead to two principles that the operating system tries to follow:
Both rules have reasonably simply understood rationale:
File systems such as NTFS work quite hard to try and satisfy the above rules. As an example, consider the case when I listen to the song “Hotel California” by the Eagles (one of my all time favorite bands). When I first save the 5MB file to my NTFS volume, the file system will try and find enough contiguous free space to be able to place the 5MB of data “together” on the disk. Since logically related data (e.g. contents of the same file or directory) is more likely to be read or written around the same time. For example, I would typically play the entire song “Hotel California” and not just a portion of it. During the 3 minutes that the song is playing, the computer would be fetching portions of this “related content” (i.e. sub-portions of the file) from the disk until the entire file is consumed. By making sure the data is placed together, the system can issue read requests in larger chunks (often pre-reading data in anticipation that it will soon be used) which, in turn, will minimize mechanical movement of hard disk drive components and also ensure fewer issued I/Os.
Given that the file system tries to place data contiguously, when does fragmentation occur? Modifications to stored data (e.g. adding, changing, or deleting content) cause changes in the on-disk data layout and can result in fragmentation. For example, file deletion naturally causes space de-allocation and resultant “holes” in the allocated space map – a condition we will refer to as “fragmentation of available free space”. Over time, contiguous free space becomes harder to find leading to fragmentation of newly stored content. Obviously, deletion is not the only cause of fragmentation – as mentioned above, other file operations such as modifying content in place or appending data to an existing file can eventually lead to the same condition.
So how does defragmentation help? In essence, defragmentation helps by moving data around so that it is once again placed more optimally on the hard disk, providing the following benefits:
The following diagram will help illustrate what we’re discussing. The first illustration represents an ideal state of a disk – there are 3 files, A, B, and C, and all are stored in contiguous locations; there is no fragmentation. The second illustration represents a fragmented disk – a portion of data associated with File A is now located in a non-contiguous location (due to growth of the file). The third illustration shows how data on the disk would look like once the disk was defragmented.
Nearly all modern file systems support defragmentation – the differences generally are in the defragmentation mechanism, whether, as in Windows, it’s a separate, schedulable task or, whether the mechanism is more implicitly managed and internal to the file system. The design decisions simply reflect the particular design goals of the system and the necessary tradeoffs. Furthermore, it’s unlikely that a general-purpose file system could be designed such that fragmentation never occurred.
Over the years, defragmentation has been given a lot of emphasis because, historically, fragmentation was a problem that could have more significant impact. In the early days of personal computing, when disk capacities were measured in megabytes, disks got full faster and fragmentation occurred more often. Further, memory caches were significantly limited and system responsiveness was increasingly predicated on disk I/O performance. This got to a point that some users ran their defrag tool weekly or even more often! Today, very large disk drives are available cheaply and % disk utilization for the average consumer is likely to be lower causing relatively less fragmentation. Further, computers can utilize more RAM cheaply (often, enough to be able to cache the data set actively in use). That together, with improvements in file system allocation strategies as well as caching and pre-fetching algorithms, further helps improve overall responsiveness. Therefore, while the performance gap between the CPU and disks continues to grow and fragmentation does occur, combined hardware and software advances in other areas allow Windows to mitigate fragmentation impact and deliver better responsiveness.
So, how would we evaluate fragmentation given today’s software and hardware? A first question might be: how often does fragmentation actually occur and to what extent? After all, 500GB of data with 1% fragmentation is significantly different than 500GB with 50% fragmentation. Secondly, what is the actual performance penalty of fragmentation, given today’s hardware and software? Quite a few of you likely remember various products introduced over the past two decades offering various performance enhancements (e.g. RAM defragmentation, disk compression, etc.), many of which have since become obsolete due to hardware and software advances.
The incidence and extent of fragmentation in average home computers varies quite a bit depending on available disk capacity, disk consumption, and usage patterns. In other words, there is no general answer. The actual performance impact of fragmentation is the more interesting question but even more complex to accurately quantify. A meaningful evaluation of the performance penalty of fragmentation would require the following:
Let’s walk through an example that helps illustrate the complexity in directly correlating extent of fragmentation with user-visible performance.
In Windows XP, any file that is split into more than one piece is considered fragmented. Not so in Windows Vista if the fragments are large enough – the defragmentation algorithm was changed (from Windows XP) to ignore pieces of a file that are larger than 64MB. As a result, defrag in XP and defrag in Vista will report different amounts of fragmentation on a volume. So, which one is correct? Well, before the question can be answered we must understand why defrag in Vista was changed. In Vista, we analyzed the impact of defragmentation and determined that the most significant performance gains from defrag are when pieces of files are combined into sufficiently large chunks such that the impact of disk-seek latency is not significant relative to the latency associated with sequentially reading the file. This means that there is a point after which combining fragmented pieces of files has no discernible benefit. In fact, there are actually negative consequences of doing so. For example, for defrag to combine fragments that are 64MB or larger requires significant amounts of disk I/O, which is against the principle of minimizing I/O that we discussed earlier (since it decreases total available disk bandwidth for user initiated I/O), and puts more pressure on the system to find large, contiguous blocks of free space. Here is a scenario where a certainly amount of fragmentation of data is just fine – doing nothing to decrease this fragmentation turns out to be the right answer!
Note that a concept that is relatively simple to understand, such as the amount of fragmentation and its impact, is in reality much more complex, and its real impact requires comprehensive evaluation of the entire system to accurately address. The different design decisions across Windows XP and Vista reflect this evaluation of the typical hardware & software environment used by customers. Ultimately, when thinking about defragmentation, it is important to realize that there are many additional factors contributing towards system responsiveness that must be considered beyond a simple count of existing fragments.
The defragmentation engine and experience in Windows 7 has been revamped based on continuous and holistic analysis of impact on system responsiveness:
In Windows Vista, we had removed all of the UI that would provide detailed defragmentation status. We received feedback that you didn’t like this decision, so we listened, evaluated the various tradeoffs, and have built a new GUI for defrag! As a result, in Windows 7, you can monitor status more easily and intuitively. Further, defragmentation can be safely terminated any time during the process and on all volumes very simply (if required). The two screenshots below illustrate the ease-of-monitoring:
In Windows XP, defragmentation had to be a user-initiated (manual) activity i.e. it could not be scheduled. Windows Vista added the capability to schedule defragmentation – however, only one volume could be defragmented at any given time. Windows 7 removes this restriction – multiple volumes can now be defragmented in parallel with no more waiting for one volume to be defragmented before initiating the same operation on some other volume! The screen shot below shows how defragmentation can be concurrently scheduled on multiple volumes:
Among the other changes under the hood in Windows 7 are the following:
Best practices for using defragmentation in Windows 7 are simple – you do not need to do anything! Defragmentation is scheduled to automatically run periodically and in the background with minimal impact to foreground activity. This ensures that data on your hard disk drives is efficiently placed so the system can provide optimal responsiveness and I can continue to enjoy glitch free listening to the Eagles :-).
Rajeev and Matt
@asesh: how would you interprete your sentence "What about a new file system that doesn't need defragmenting? When will see such a file system in Windows? Linux already has one. We had high hopes from WinFS and hopefully when it's done it will remove the need to defragment our hard disks from time-to-time." other than that you call WinFS as file system.
Can you give some or at least one technical example to show us how Unix based systems are more secure than Windows systems which are based on an NT kernel?
I don't see how using an OS gives you clues on the security level of an OS.
I wish the file system could treat expandable files differently than static files. Much of the fragmentation on an aged system is from files that have been fragmented due to them being expanded. Files that are frequently expanded need to have a lot of extra room at the end of the file, where static files (ie: dlls) are never expanded and can be packed tight. I believe that Raxco recognized this more than 15 years ago on VMS.
@sirus It's a good point. For a typical user defrag should happen during disk idle times and maybe share that idle time with desktop search service. Defrag should happen only when on AC power.
Also there should be less HDD thrashing during idle time because HDDs seems to be TOO loud when building search indexes. That happens because because Windows architects think that they can put 100% load on disk during disk idle time. They did not think about all the noise they create with this. Especially at night!!
The silly idea that most users should know how to tweak their systems should go away. I liked to watch defragmentation status screens since Norton Speedisk for DOS but that time has gone. I don't care anymore.
I know many professional C++ devs who don't care about installing updated drivers in spite of having occasional BSODs. And they would not run defrag, they would just use Macs at home.
I am sure that PMs for the Defrag feature understand all that well but are not brave enough to say it out loud and then do something to implement. (Yes, I am lazy too).
@Asesh I am sorry about that but statistically Windows Vista seems to be more secure and reliable than Unix systems. Reliable - in terms of OS. Not hardware.
Apple has full control on the hardware and it does a lot of testing of only that specific combination of hardware. MS will never achieve that in a PC. For instance, my Nvidia _Business_ Platform motherboard had Vista stickers all over it but it has faulty Ethernet driver that hangs system when I unplug network cable (independently of what the cable is connected to from the other side).
"The design of Unix is very old (and IMHO outdated)."
I hear this alot being familiar with Unix, Unix-Like OS's and Windows i just have to LOL. I laugh the same at the old Unix battle axes that think Windows is not a Serious Server OS and is unstable. From my experience 99% of Windows instability comes from 3rd party poor software design/coding; companies not following clear guidelines on how to develop for Microsoft OS’s.
In regards to defrag I still miss the Norton Disk Doctor defrag interface from the DOS days or even win98(please consider bringing this view back MS!)it was mesmerizing watching it stack the blocks.
Anyway thanks for the laugh.
First, great article and nice work!
Now, my issue here does not have to do so much with the way Windows handles fragmented files and I/O, it's the implied state the end user is asked to leave their computer in...On
Leaving your computer on 24/7 is an absurd waste of power/energy and is, in my opinion, plain irresponsible in this day and age.
I feel it would be a huge step forward and gesture on Microsoft's behalf if they were to either move to a more efficient way to concurrently handle defragging in the background while performing common computing (ie- while machine is idle) to eliminate the need for scheduled defragmenting in the first place. Or if they promoted responsible use by NOT having the schedule default to 1 am and explain to the user in some way so they can be educated as to why.
I realize it's a stretch, but a top down approach to helping to solve a global crisis would be HUGE. And this seems like such a relatively easy thing to implement imo.
I've noticed much the same thing having spent the past month or two experimenting with defragging - much of the fragmentation comes from files that it wouldn't be difficult for Windows to "know" are going to become fragmented - Web browser caches are particularly bad for this. Maybe having a look at the layout rules for NTFS again might not be a bad idea.
The defragment option prior to Vista was a great tool for incompetent IT help desk techs to use for problems that could not be solved with their number one solution of rebooting the computer.
Disk defragmentation would take hours to complete, and had pretty pictures to show that the computer was doing something. With any luck the user would give up asking for help by the time the defrag was complete and the ticket could be closed.
Thanks everyone for your comments so far, we’ve noticed several common questions that we’d like to answer:
@ari9910/Vyacheslav Lanovets/nicbot: The default scheduled run time was picked to avoid interfering with interactive usage. These defaults, of course, can be changed. If defrag is unable to run at the scheduled time because the computer is not on, it will then automatically be scheduled to run next when the computer is idle.
@Southpaw42: The drawback to defragmenting at creation or update time is that it adds potentially significant latency and I/O overhead to the operation, especially if the create/update is blocked from completing until the defrag is complete. For Windows, that would not be an appropriate design as it would tradeoff system responsiveness, which is very important, in favor of decreasing fragmentation, a relatively less valuable objective.
@tgrand: There are several reasons for disabling defrag on SSDs – keep in mind that SSDs are a relatively recent technology. Our internal evaluation of SSDs demonstrated that there’s a significant amount of variability in delivered performance. While there are possible benefits to defragmenting SSDs (such as coalescing free space and being able to issue I/O in larger chunks), we were concerned about the potential of decreasing the life time of the flash media from additional I/O. We determined that prioritizing media lifetime and ensuring reliability of data was the correct choice. As expected, we will continue to actively monitor and test this new media type to ensure we optimize our behavior appropriately.
Hope you've had a chance to try out the Win7 Beta!
File and Block Storage Team
The defrag tool is much more improoved in Windows 7 from Vista! It it some functions that i miss in the defrag tool:
- Defrag on boot (Defrags also system and reistry)
- A diagram that shows the current filestructure on the HDD we are defragmenting
- A progressbar
- A field which shows what files that are being defragmented
I like the new design, but it could be improoved.
when you say
"The default scheduled run time was picked to avoid interfering with interactive usage. These defaults, of course, can be changed. If defrag is unable to run at the scheduled time because the computer is not on, it will then automatically be scheduled to run next when the computer is idle"
do you say that this is the default behavior of the defrag task or do you say that the user can (must) reconfigure it to work like that?
I just checked my defrag task. The last time should have been 28.1. on 1 am. The system (Windows 7 beta 1 x86) was off at that time. Yesterday and today it was at least idle for 40-60 minutes when I went to launch. But defragging was not started. The last time shown in the task scheduler was my manual defrag action last week.
What should I expect here? Does the defrag task not update the "last run time" when it is started but does not find a drive worth defragging? Or is this a scheduling bug?
Thanks for the reply, Matt! That's exactly the kind of explanation that I would hope to see when someone talks about defrag and SSDs. I understand and agree with your approach. I just think that sooner or later, people will find out that SSDs are not magically immune to fragmentation, and they'll appreciate having a better understanding of the situation. Admittedly, it's a pretty complex picture.
It will be very interesting to see how this issue evolves over time.
Alright, seriously, this 1AM auto-scheduler is simply a flaw at all levels. First, because almost nobody lets their computer open all night. Secondly, because EVEN if it's open at 1AM, it's probably because i'm working on it. Third, because EVEN if it reschedule automatically another day, it will still have problem #2.
I think, like most people said here, the best way would be to automatically start it when the I/O is for some time (like while i'm browsing for 2 hours) and make the defrag able to start and stop quite rapidly.
The same thing applies to windows updates.
It would be nice is the system would consider the file type and place them on a pre-reservered place on a harddisk.
To make my point:
An .txt or .log is usually a lot smaller then a .MP3, and a MP3 again is usually a lot smaller then a .AVI.
System files would never grow, until they got replaced/upgraded.
I believe for some file types you could define a place on the harddrive so they don't get all over the place.
It seems like the algorithm for defragmentation itself must be very poor. Even on an otherwise idle system, with >50% disk space, it takes far longer to defragment than 2 full reads and writes of the data would explain. An order of magnitude more at least. Is any attention being given to actually trying to make defragmentation faster?
@SvenC/neken: Let me explain further how the scheduler works. Defrag won't actually run at 1AM, unless the machine is idle at that time and, if you're using a laptop, not on battery power. (Conserving battery power is an important goal for Win7 and we've made changes to support this throughout the system.) In addition, if defrag is running and you start to use the system, defrag will stop until the system becomes idle again at which point it will resume.
Quite a bit of work went into intelligently detecting idle time and interactive use. You can learn more at the following links:
PDC - http://channel9.msdn.com/pdc2008/PC19/
Paper - Designing Efficient Background Processes: http://www.microsoft.com/whdc/system/pnppwr/powermgmt/BackgroundProcs.mspx
File and Block Storage Team