Disk Defragmentation – Background and Engineering the Windows 7 Improvements

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

ari9910 — Mon, 26 Jan 2009 05:12:24 GMT

Has it ever occurred to anyone on the team that the best time to run things like the defragmenter, updater, and backup programs are when we AREN'T using the system. as said in the blog, it won't run if we turn the system off every night. So if those things ran when we weren't using the computer yet it was still on, those operations would run and we wouldn't have to worry about it. I find it increasingly annoying that the updater wants to update (and annoy me about it) while i'm using it because i shut the computer off at the 3:00AM time it is set to run. Similar goes for the defragmenter. I install 3rd party programs because of that, i would appreciate if I didn't have to do that because, i personally love Microsoft and hope that one day it will actually think more about customer satisfaction and less about profits (to a reasonable degree of course).

New file system that doesn't need defragmenting

Asesh — Mon, 26 Jan 2009 06:48:55 GMT

What about a new file system that doesn't need defragmenting? When will see such a file system in Windows? Linux already has one. We had high hopes from WinFS and hopefully when it's done it will remove the need to defragment our hard disks from time-to-time.

Please don't release the next version of Windows (after Windows 7) until WinFS is done and a UNIX based kernel or a better one is implemented because Windows is still prone to viruses unlike Mac OS X and Linux which are more secure than Windows.

I hope all the Windows' coders out there will see this because it's high time so we need such a file system and kernel. Thanks

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

mark_ms — Mon, 26 Jan 2009 06:51:17 GMT

I've had a keen interest in defrag products since the days of the 286 and 40Mb hard drives. I find it particularly refreshing to learn about where the line crosses between useful defragging and wasted effort and how much newer technology has been minimizing the effect of fragmentation. Now that I have a better understanding of how Windows 7 defrags, I am now even less concerned about managing it myself.

I really pity the 3rd party defragmenters out there. They now have a higher bar to clear to convince me that the default defragger isn't good enough.

I don't really mind the defragger kicking in when I'm doing low I/O work e.g. browsing the web but from the screenshots, it seems that this has to be scheduled. I can't always know what kind of work I would be doing and am reluctant to specify a time when I know the computer would be on and I am working on it. So, I would prefer that there is an option for the defragger to kick in during low I/O if it hasn't been getting a chance to run at the normal scheduled time.

One thing I do is recompress tv recordings. Since my recompressor is currently single-threaded, I am very much tempted to run two at the same time on my dual-core machine. Would this create much more fragmented files than if I run them one at a time?

Please Start Thinking Outside of the Box

obsidience — Mon, 26 Jan 2009 07:24:08 GMT

First off, I'm delighted that tool disk defragmentation tool was deemed important enough to be blogged on. Kudos!

But if you want to improve the speed and efficiency of this new OS then you need to think about:

1) A defrag tool that can be scheduled to run outside of windows when no files are locked by the OS. This would allow for a fully defragged disk.

2) A defrag tool that can profile a standard computer session and defrag files accordingly. Moving heavily accessed files to the outer areas of the disk platter where data density to rotational speed is highest

3) A defrag tool that analyzes the boot up pattern and can place files sequentially along the zone allowing for less access and more reads.

If you investigated these ideas I'm sure you can further reduce your boot up times by a substiantial amount.

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

SouthPaw42 — Mon, 26 Jan 2009 07:45:41 GMT

Wouldn't a file system that included a process to reduce fragmentation during file creation and update. I have an OS HDD that is 60% empty space but after 30 days is 50% fragmented. A drive that empty should not be fragmented.

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

Vyacheslav Lanovets — Mon, 26 Jan 2009 08:49:14 GMT

I believe that scheduling logic must be changed too. People who do not understand what defrag is tend to turn off their PCs early. Windows MUST detect that defrag did not have a chance to run for a month or two and advise changing scheduled time accordingly or even do it silently according to user behaviour pattern. But this will be too user-friendly for Microsoft :)

For advanced users Vista defrag can be scheduled to run at boot time, and in this mode it seems to defrag more than during user session. And reboot can be scheduled too. And sometimes even hardware wakeup can be scheduled :)

As Vista itself, Vista defrag is already good but it seems like scheduling logic did not change i Windows 7. Kind of: it won't work for my mom who at the same time really needs defrag and does not leave your PC turned on at 1:00 AM!

Windows 8 or 9?

re: New file system that doesn't need defragmenting

II ARROWS — Mon, 26 Jan 2009 11:44:13 GMT

Asesh, "file system that doesn't need defragmenting" simply doesn't exist.

You wrote "Unix", those file systems work defragmenting while writing new file or expanding them. The results are slower writing time, in most case if a file is wroten you are working and this slow your work. Windows perform this while you are not working.

Think about converting a video, this operation require a large amount of IO operation in RAM and in disk, generating a lot of file that have to be deleted at the end of the process. If the file system have to defrag while writing, conversion time would be increased.

Also, you wrote "WinFS". WinFS was not a file system, but an added layer that works on NTFS, an heavy one.

An idea on wich WinFS was built is splitting up the meta data from data, and in this article I see another step closer to WinFS.

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

sirus — Mon, 26 Jan 2009 12:45:44 GMT

@ Asesh

A file system which has the characteristic you mention doesn't exist. If you take Linux and its most used file system, which is EXT3, the fragmentation percentage is indeed low, however it's calculated only on big chunks, just like on Windows Vista. If you try another file system such as XFS which is certainly faster than EXT3 you'll notice that it suffer a greater fragmentation and due to that XFS is provided with a de-fragmentation tool.

Ending the off-topic, I'd like an "intelligent" de-fragmentation process that runs when I'm not heavily using my HD and possibly an "intelligent" scheduler which automatically modify its configuration based on users' habits.

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

Eiki — Mon, 26 Jan 2009 12:59:00 GMT

What about SSD? How does this defragmentation process change if there is an SSD in place?

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

RobertWrayUK — Mon, 26 Jan 2009 13:17:19 GMT

@Eiki

One of the bullet points at the end of the entry:

"If solid-state media is detected, Windows disables defragmentation on that disk. The physical nature of solid-state media is such that defragmentation is not needed and in fact, could decrease overall media lifetime in certain cases."

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

Domenico — Mon, 26 Jan 2009 13:41:55 GMT

My computer is always on With Vista or Windows 7.

automatically defrag form me is very interesting and work fine,

in the Beta of Windows 7, Defrag is even better.

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

d_e — Mon, 26 Jan 2009 13:47:12 GMT

@Asesh: It seems you have no clue about kernels and filesystems. Especially the windows kernel.

1. WinFS is not a filesystem. It's a database. NTFS runs underneath.

2. There is not filesystem which doesn't fragment. Such a thing is impossible.

3. "UNIX or better". What do you mean by this? Unix-compatible systems aren't inherently more secure than Windows. The design of Unix is very old (and IMHO outdated). I'm confident that Microsoft is wise enough to keep their current kernel.

4. A secure kernel doesn't give you a secure computer. Because most users aren't computer literate and will gladly install anything if an email tells them to do so.

To the article: I believe defragmentation is misunderstood by most users. Many think this is some sort of magic powder that will make their machine much faster. It won't. Users shouldn't have to worry about defragmentation. The decision to hide all this in Vista was the right one. You guys get the 100+ emails from self-proclaimed experts which (mostly) have no clue what they're writing about.

The only thing I'm wondering is why my disk (according to the HDD LED) works only 50% when resuming from hibernate...

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

II ARROWS — Mon, 26 Jan 2009 14:11:40 GMT

I've a question about my test configuration:

I've an HDD with Vista x64 and 7 x64, another disk used for data.

I always move Document(and music, picture and video) folder on the second disk to preserve them if I need to format or simply change machine.

The structure is like XP:

My Documents

|----My Music

|----My Pictures

|----My Videos

After set in the library the folders and set as default save directories, if I boot Vista and try to access the Documents sub folder(Only music, pic and video) it alerts me that I need to be the owner. Only for those folder, files in My documents can be opened and modified.

May it depends because of the new "intelligent defrag" splitting meta data, and 7 doesn't want Vista to "undo" 7's work?

Or is just a bug that 7 wants those right?

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

Asesh — Mon, 26 Jan 2009 17:45:33 GMT

@d_e

Read it carefully, I didn't say WinFS is a file system. And unix based kernel is more secure than Windows kernel :P Yes I do use both Linux and Mac OS X so I can say that.

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

tgrand — Mon, 26 Jan 2009 18:41:57 GMT

"The physical nature of solid-state media is such that defragmentation is not needed..."

Did you guys check this or just take some SSD manufacturers word for it? It's really disappointing to hear this from you. I was skeptical about this claim so I tested it myself. It wasn't hard at all to get a 30% performance hit reading a fragmented file from an SSD. Sure, that's not nearly as bad as it would've been on a spinning disk, but 30% is still significant - especially when you're paying a huge price premium to use one of these drives for performance reasons!

I argued this point with the manufacturer and they eventually conceded, but said the performance benefit of defragging was outweighed by the shortening of the drive's lifespan. I said "OK great, you should claim this, instead of claiming that it's 'not needed', which is a half-truth at best." As you might imagine, they weren't very receptive to that idea. And I only got to do some limited testing. I'm sure things get much worse when you have a nearly full system drive used over a long time.

By the way, here's why fragmentation hurts on an SSD despite the minimal "seek time" penalty: most SSDs only get their high throughput when the individual I/O requests are for sufficiently large amounts of data. Fragmentation can easily turn a file read operation from a handful of large fast reads into hundreds of tiny slow ones.

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

SvenC — Mon, 26 Jan 2009 22:18:07 GMT

@asesh: how would you interprete your sentence "What about a new file system that doesn't need defragmenting? When will see such a file system in Windows? Linux already has one. We had high hopes from WinFS and hopefully when it's done it will remove the need to defragment our hard disks from time-to-time." other than that you call WinFS as file system.

Can you give some or at least one technical example to show us how Unix based systems are more secure than Windows systems which are based on an NT kernel?

I don't see how using an OS gives you clues on the security level of an OS.

SvenC

It should treat expandable files differently than static files

shan_mcarthur@spamcop.net — Mon, 26 Jan 2009 23:53:33 GMT

I wish the file system could treat expandable files differently than static files. Much of the fragmentation on an aged system is from files that have been fragmented due to them being expanded. Files that are frequently expanded need to have a lot of extra room at the end of the file, where static files (ie: dlls) are never expanded and can be packed tight. I believe that Raxco recognized this more than 15 years ago on VMS.

Shan

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

Vyacheslav Lanovets — Tue, 27 Jan 2009 00:21:33 GMT

@sirus It's a good point. For a typical user defrag should happen during disk idle times and maybe share that idle time with desktop search service. Defrag should happen only when on AC power.

Also there should be less HDD thrashing during idle time because HDDs seems to be TOO loud when building search indexes. That happens because because Windows architects think that they can put 100% load on disk during disk idle time. They did not think about all the noise they create with this. Especially at night!!

The silly idea that most users should know how to tweak their systems should go away. I liked to watch defragmentation status screens since Norton Speedisk for DOS but that time has gone. I don't care anymore.

I know many professional C++ devs who don't care about installing updated drivers in spite of having occasional BSODs. And they would not run defrag, they would just use Macs at home.

I am sure that PMs for the Defrag feature understand all that well but are not brave enough to say it out loud and then do something to implement. (Yes, I am lazy too).

@Asesh I am sorry about that but statistically Windows Vista seems to be more secure and reliable than Unix systems. Reliable - in terms of OS. Not hardware.

Apple has full control on the hardware and it does a lot of testing of only that specific combination of hardware. MS will never achieve that in a PC. For instance, my Nvidia _Business_ Platform motherboard had Vista stickers all over it but it has faulty Ethernet driver that hangs system when I unplug network cable (independently of what the cable is connected to from the other side).

@d_e

smartpatrol — Tue, 27 Jan 2009 00:53:04 GMT

"The design of Unix is very old (and IMHO outdated)."

I hear this alot being familiar with Unix, Unix-Like OS's and Windows i just have to LOL. I laugh the same at the old Unix battle axes that think Windows is not a Serious Server OS and is unstable. From my experience 99% of Windows instability comes from 3rd party poor software design/coding; companies not following clear guidelines on how to develop for Microsoft OS’s.

In regards to defrag I still miss the Norton Disk Doctor defrag interface from the DOS days or even win98(please consider bringing this view back MS!)it was mesmerizing watching it stack the blocks.

Anyway thanks for the laugh.

How about a more current approach?

nicbot — Tue, 27 Jan 2009 01:14:12 GMT

First, great article and nice work!

Now, my issue here does not have to do so much with the way Windows handles fragmented files and I/O, it's the implied state the end user is asked to leave their computer in...On

Leaving your computer on 24/7 is an absurd waste of power/energy and is, in my opinion, plain irresponsible in this day and age.

I feel it would be a huge step forward and gesture on Microsoft's behalf if they were to either move to a more efficient way to concurrently handle defragging in the background while performing common computing (ie- while machine is idle) to eliminate the need for scheduled defragmenting in the first place. Or if they promoted responsible use by NOT having the schedule default to 1 am and explain to the user in some way so they can be educated as to why.

I realize it's a stretch, but a top down approach to helping to solve a global crisis would be HUGE. And this seems like such a relatively easy thing to implement imo.

Regards.

@ Shan

Hairs — Tue, 27 Jan 2009 01:19:15 GMT

I've noticed much the same thing having spent the past month or two experimenting with defragging - much of the fragmentation comes from files that it wouldn't be difficult for Windows to "know" are going to become fragmented - Web browser caches are particularly bad for this. Maybe having a look at the layout rules for NTFS again might not be a bad idea.

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

gss4w — Tue, 27 Jan 2009 01:21:21 GMT

The defragment option prior to Vista was a great tool for incompetent IT help desk techs to use for problems that could not be solved with their number one solution of rebooting the computer.

Disk defragmentation would take hours to complete, and had pretty pictures to show that the computer was doing something. With any luck the user would give up asking for help by the time the defrag was complete and the ticket could be closed.

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

mgarson — Thu, 29 Jan 2009 03:07:56 GMT

Thanks everyone for your comments so far, we’ve noticed several common questions that we’d like to answer:

@ari9910/Vyacheslav Lanovets/nicbot: The default scheduled run time was picked to avoid interfering with interactive usage. These defaults, of course, can be changed. If defrag is unable to run at the scheduled time because the computer is not on, it will then automatically be scheduled to run next when the computer is idle.

@Southpaw42: The drawback to defragmenting at creation or update time is that it adds potentially significant latency and I/O overhead to the operation, especially if the create/update is blocked from completing until the defrag is complete. For Windows, that would not be an appropriate design as it would tradeoff system responsiveness, which is very important, in favor of decreasing fragmentation, a relatively less valuable objective.

@tgrand: There are several reasons for disabling defrag on SSDs – keep in mind that SSDs are a relatively recent technology. Our internal evaluation of SSDs demonstrated that there’s a significant amount of variability in delivered performance. While there are possible benefits to defragmenting SSDs (such as coalescing free space and being able to issue I/O in larger chunks), we were concerned about the potential of decreasing the life time of the flash media from additional I/O. We determined that prioritizing media lifetime and ensuring reliability of data was the correct choice. As expected, we will continue to actively monitor and test this new media type to ensure we optimize our behavior appropriately.

Hope you've had a chance to try out the Win7 Beta!

Matt Garson

File and Block Storage Team

Microsoft

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

martinmine — Thu, 29 Jan 2009 15:40:37 GMT

The defrag tool is much more improoved in Windows 7 from Vista! It it some functions that i miss in the defrag tool:

- Defrag on boot (Defrags also system and reistry)

- A diagram that shows the current filestructure on the HDD we are defragmenting

- A progressbar

- A field which shows what files that are being defragmented

I like the new design, but it could be improoved.

Martin

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

SvenC — Thu, 29 Jan 2009 17:35:49 GMT

Hi Matt,

when you say

"The default scheduled run time was picked to avoid interfering with interactive usage. These defaults, of course, can be changed. If defrag is unable to run at the scheduled time because the computer is not on, it will then automatically be scheduled to run next when the computer is idle"

do you say that this is the default behavior of the defrag task or do you say that the user can (must) reconfigure it to work like that?

I just checked my defrag task. The last time should have been 28.1. on 1 am. The system (Windows 7 beta 1 x86) was off at that time. Yesterday and today it was at least idle for 40-60 minutes when I went to launch. But defragging was not started. The last time shown in the task scheduler was my manual defrag action last week.

What should I expect here? Does the defrag task not update the "last run time" when it is started but does not find a drive worth defragging? Or is this a scheduling bug?

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

tgrand — Thu, 29 Jan 2009 19:35:52 GMT

Thanks for the reply, Matt! That's exactly the kind of explanation that I would hope to see when someone talks about defrag and SSDs. I understand and agree with your approach. I just think that sooner or later, people will find out that SSDs are not magically immune to fragmentation, and they'll appreciate having a better understanding of the situation. Admittedly, it's a pretty complex picture.

It will be very interesting to see how this issue evolves over time.

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

Neken — Fri, 30 Jan 2009 05:42:16 GMT

Alright, seriously, this 1AM auto-scheduler is simply a flaw at all levels. First, because almost nobody lets their computer open all night. Secondly, because EVEN if it's open at 1AM, it's probably because i'm working on it. Third, because EVEN if it reschedule automatically another day, it will still have problem #2.

I think, like most people said here, the best way would be to automatically start it when the I/O is for some time (like while i'm browsing for 2 hours) and make the defrag able to start and stop quite rapidly.

The same thing applies to windows updates.

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

sokolum — Fri, 30 Jan 2009 12:55:37 GMT

It would be nice is the system would consider the file type and place them on a pre-reservered place on a harddisk.

To make my point:

An .txt or .log is usually a lot smaller then a .MP3, and a MP3 again is usually a lot smaller then a .AVI.

System files would never grow, until they got replaced/upgraded.

I believe for some file types you could define a place on the harddrive so they don't get all over the place.

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

Surt — Fri, 30 Jan 2009 21:02:32 GMT

It seems like the algorithm for defragmentation itself must be very poor. Even on an otherwise idle system, with >50% disk space, it takes far longer to defragment than 2 full reads and writes of the data would explain. An order of magnitude more at least. Is any attention being given to actually trying to make defragmentation faster?

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

mgarson — Sat, 31 Jan 2009 04:25:30 GMT

@SvenC/neken: Let me explain further how the scheduler works. Defrag won't actually run at 1AM, unless the machine is idle at that time and, if you're using a laptop, not on battery power. (Conserving battery power is an important goal for Win7 and we've made changes to support this throughout the system.) In addition, if defrag is running and you start to use the system, defrag will stop until the system becomes idle again at which point it will resume.

Quite a bit of work went into intelligently detecting idle time and interactive use. You can learn more at the following links:

PDC - http://channel9.msdn.com/pdc2008/PC19/

Paper - Designing Efficient Background Processes: http://www.microsoft.com/whdc/system/pnppwr/powermgmt/BackgroundProcs.mspx

Matt Garson

File and Block Storage Team

Microsoft

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

wtroost — Sat, 31 Jan 2009 06:59:37 GMT

Just writing to agree with sokolum: "It would be nice is the system would consider the file type and place them on a pre-reservered place on a harddisk."

On a side note, much of the performance loss people complain about is realited to Explorer add-ons (not disk defrag.) Any chance for an Explorer add-on manager of kinds? Run em in a different process please!

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

anonymuos — Sat, 31 Jan 2009 07:10:55 GMT

Can you make it such that when multiple volumes are selected, they can defragged one after the other (not in parallel) from the GUI. This makes it use defrag.exe and give up on the nice GUI. Defragmenting multiple volumes simultaneously takes a performance hit if I'm doing something else too on my PC.

Disk Defragmentation – Slow as Grandma Moses

graham.lv — Sun, 01 Feb 2009 13:02:43 GMT

I didn't see this - guess I've been busy using 7. Anyway I just recently sent feedback that it's useless. Got a Samsung 160GB Sata II small HD on my test machine and have used about 52 GB.

Ran Defrag and in just puts 'pas 1 0.5%' and up to 100%, then it puts 'pass 2 35%, etc - apparently there are 10 passes - I didn't stop to watch, I watered the garden, did the washing and cooked a meal and it was still going... and going... after 4 hours it was on about 'pass 10 .05%'.

There are commercial defragers that do a little bit in the background - or - they may be bull..

Whichever way - over 4 hours of not touching the computer to Defrag 52 GB means that Defrag is totally USELESS and will never work and/or run.

160 GB is the smallest HD I could buy - most people will be buying 2 TB. 3 months to Defrag????

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

Mattisdada — Sun, 01 Feb 2009 15:21:48 GMT

I was under the impression that the performance boost of defraging a SSD is like only 0.5%. As the access time is virtually 0 on a SSD as it uses flash memory rather then a spinning platter, so it didnt matter as much if they were a bit muddled.... It could find them very quickly. A small overhead at most..... not worth the loss of lifetime and reliability.

And the whole Windows Kernel(vs unix) is unsecure argument going on.... Is just insane. Obviously the Winows Kernel is more secure....

Ill try and explain it metaphysically. There is two towns in the ye old medievil days. One town is a large town of millions (windows users), its heavily fortified, theres always attackers trying to take over its lands and get inside and kill the people inside. Every now and then one does get in and gives a few people the plague, but they have pretty good doctors and they kill that type of plague.

Now town 2. Well its a small little forrest village. Only has about 500 people. Very community like village. They have no riches. They have nothing. No one attacks them. But the big rich city always has people trying to attack it...

So although at times Windows may seem unsecure, its EXTREMELY secure relatively. Its just a matter of perspective. Oh and sorry if my story sucked on trying to explain it to people who blatantly don't understand :). And most unstableness is indeed 3rd party. When Vista came out. It was like 70% of crashes were caused by nVidia, 5% ATI, 22% others, 3% Microsoft. Or something like that, i cant remember the specifics. But it was a mostly nVidias fault scenario.

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

tgrand — Mon, 02 Feb 2009 18:50:02 GMT

Mattisdada:

That is a very common misconception, propagated by false claims from SSD manufacturers, software vendors, reviewers, etc. People use the relatively low access time of SSDs to convince themselves and others that fragmentation is a non-issue. But really, there are two main factors that determine how long it takes to read a file from start to finish: 1) access time (a time cost paid per I/O request) and 2) throughput (how much data can be read per unit of time).

On a traditional spinning disk, the relatively high access time is what causes the most slowdown when trying to read a fragmented file. If you need to read a file which is in 200 fragments, and your seek time is an average of 9ms, that's 1.8 seconds spent just seeking around. On an SSD, that might be more like 200 * 0.1ms = 20ms. A 90x improvement.

But access time is only part of the picture. The other part is throughput, and it's very important. Let's say your SSD is capable of reading at 200MB/sec. Well guess what - its throughput of an SSD is actually quite variable depending on I/O size. The size of a single I/O request affects the throughput you get. Look at the graphs here:

http://www.guru3d.com/article/gskill-ssd-solid-state-disk-64-gb-review/6

If you have to issue a bunch of I/O requests for say, 64KB and under, you're looking at 2-10x decrease in throughput for those requests. The more fragmented a file is, the more small I/O requests are going to be needed to read it. It will not be as fast as reading a contiguous file using larger I/O requests, and the difference could easily be much more than 0.5%.

As I said before, I measured a 30% hit myself. In that particular test, I used XP, an SATA2 128GB MLC SSD, a 180MB file in 200 fragments (downloaded by Firefox), SysInternals contig to measure and remove fragmentation, filemon to check actual I/O sizes, and a program I wrote to do read timing using different APIs, flags, and requested I/O sizes.

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

adir1 — Tue, 03 Feb 2009 21:14:44 GMT

Nice summary of graphs and charts in the beginning, but I feel like big "feature" is missing here.

Perhaps it is not directly related to disk fragmentation (though I though I heard it was back in early Vista days), but what about "aligning" of software for faster loads.

Great example is during system boot - the OS knows it will need a lot of drivers, registry, executable files (dlls, etc). Aligning all those for one or very few contiguous load would significantly improve startup.

Granted this is something that can be "prepared" during the initial OS setup, but overtime as you add hardware, update kernel pieces (security, anyone?), what will re-align these pieces for fast load?

Plus, what about Icons and Background graphics and other such "nonsense" - this is all part of "creating" the user desktop, and the faster it happens the better the experience!

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

adir1 — Tue, 03 Feb 2009 21:24:50 GMT

One more comment, to add my 2 cents to what tgrand mentions above:

1. The SSD drives are in their infancy, and manufactures (especially Intel, I hear), are making huge leaps forward with the way data is internally organized to provide huge seek/throughput improvements

2. Due to the nature of Flash memory Writing and Re-writing to same memory regions degrades media at an accelerated rate. To me, that says that a proper place to "defragment" a file is inside internal firmware of the SSD drive, and not externally by OS. Anyway, I understand most firmware already makes these decisions of where to physically write data, separate from the "logical" OS positioning, based in part on media degradation optimizations.

On last note, I'd like to see a post about what we're doing to optimize OS for SSD drives. There is a world of functionality and improvements that can be gained, way beyond the silly "Boost" or whatever that thing is called in Vista. I am talking about scenarios where boot partition is SSD, or it's other kind of "mix" where system contains SSD and old-school drives. Another post, Win 7 team, perhaps?

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

tgrand — Wed, 04 Feb 2009 02:00:17 GMT

adir1:

You said "the proper place to 'defragment' a file is inside internal firmware of the SSD drive." But how could this be done? The kind of fragmentation we're talking about here occurs at the filesystem level - the "logical" OS positioning as you called it. Only the OS can manage the filesystem. A storage device can't possibly do it.

It sounds like you're either mixing the concepts of filesystem fragmentation and wear leveling (they're really completely separate), or you're suggesting there should be some kind of new and very different interaction between OS and storage device...?

Fixing the problem

Rudi Larno — Wed, 11 Feb 2009 21:59:04 GMT

I must agree with adir1, solokum, hairs and shan.

How windows organizes the disk is far more relevant than the old-school 'defragmentation' routine. The first graph shows that disks are too slow compared to the CPU. So why not use the cpu power to determine the most optimal place for a file when it is written? And use a continuously running service to optimize the disk when not in use. (like diskeeper)

I've written more of how I'd like to see windows organize the disk @ http://www.larud.net/subtext/archive/2009/02/10/46.aspx

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

Mr32bit — Sat, 21 Feb 2009 02:22:25 GMT

I would add this to the Disk Defrag option.

Have an option of allowing disk defrag during a screen saver. This would help out since it is during an idle time. Make it a default and allow it able to be turned off.

I like the other suggestions provided too, but I think it might hinder performance if the OS is continually monitoring when "idle" time is available.

Disk defrag during a screen saver is a good option. Although not everyone turns their screen saver on, it would be helpful.

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

DWalker — Thu, 26 Feb 2009 06:21:20 GMT

Mr32bit, I think the OS always knows when the computer is idle. It's not that hard to detect.

And, screen savers (and screen power-off) on laptops that are running on battery power are there to save the battery; defrag while running on a bettery may not be ideal.

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

RikDederly — Fri, 27 Feb 2009 00:23:16 GMT

I have mixed feelings about the following suggestions, but these are ideas I had while reading the post.

1) Schedule a hardrive to turn on at a specified time and turn off once the defrag was complete. The drawback for this would be the additional power used in the middle of the night. The benefit would be a defragmented drive at little impact to the user. Of course, there would be no connection to a network or internet when this occurs.

2) Create an automatic defrag to occur when the system has been idle for 3-4 hours. Under normal use, this would only occur when your system has been left on overnight. Therefore, the user would only need to leave the system on during any given night. The current scheduled listed above appears to require the user to remember to leave the system on on a given day (such as Wednesday). Many users would likely forget to leave their system on during their scheduled defrag time.

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

Victor Dubina — Fri, 27 Feb 2009 12:56:42 GMT

To Scheduler automatic defragmentation on idle just simply go to Control Panel > Administrative Tools > Task Scheduler. Expand Task Scheduler Library > Microsoft > Windows. Find "Defrag" subfolder, right click on "ScheduledDefrag" task, choose "Properties" from falling down menu. Click "Triggers" tab.

Here you may add a new trigger. A limit is only your imagination :) For instance, choose "On idle" from "Begin a task:" list. Press Ok or tune-up with advanced settings.

As simple as that.

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

theophoretos — Mon, 30 Mar 2009 22:04:55 GMT

Something good that I just read about windows 7 is that "termination of defragmentation would not damage the system". But does this mean that it would on Vista? I mean, today I started defragmenting my vista HD for the first time, when it has only 10 gb left on a 250 gb HD. After more than an hour, I noticed that defrag.exe was not even consuming any CPU at all on the task manager and assumed that it was all done, and so I killed the process. Only then did I notice it was still going on before I killed it. Would this have damaged any data on my drive?

Having a status report while defrag is in process now has another reason: so you can actually tell that it is still going on!

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

tgrand — Mon, 30 Mar 2009 23:48:24 GMT

I believe the defrag API and implementation in modern Windows is set up so that it shouldn't be possible to have data loss or corruption as a result of interrupting the defrag process - whether the interruption is you killing a process, a driver causing a BSOD, power to the system being cut, etc. I highly doubt there was any fundamental change here between Vista and Windows 7. But it would've been nice if you'd named your source.

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

graziano — Wed, 01 Apr 2009 02:59:31 GMT

I read somewhere that the defragmenter in Vista can't defragment files over 64MB and doesn't include these files in the list of top fragmented files when you run the analysis from the DOS prompt. Is this true and does this limitation still exist in the Windows 7 defragmenter?

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

derosnec — Mon, 29 Jun 2009 00:04:46 GMT

Not everything gets defragged. Some system files are immovable. See:

http://support.microsoft.com/kb/227350

(Files Excluded by the Disk Defragmenter Tool)

re: Disk Defragmentation – Background and Engineering the Windows 7 Improvements

Bertrand2 — Mon, 03 Aug 2009 09:57:57 GMT

@obsidience:

Your "zones" proposal does sound like a good idea, but I think it is probably bad both in theory and in practice.

In theory, it falls foul of a few principles:

1. Code optimization is an empirical science. You have not measured the cost of not using the zone system. Of course, this is overcome with a fair bit of work. But as far as we know, your proposal may not deliver much in the way of savings, but will create a number of complications.

2. If you don't want a radical redesign of either the filesystem or the Windows API, you will need to have the defragmenter make assumptions about the semantic content of the files. This is all kinds of bad. First of all, it will probably be wrong. Even if you get it right, it will become wrong with changes to the API. This sort of thing was the number 1 cause of bugs in earlier Windows OSes and their applications. See Raymond Chen's 'The Old New Thing' blog for various rants. It used to be a little more acceptable, because without those optimizations Windows would have run so slowly as to be unusable, but today it is Really Wrong.

3. Radically changing the API to optimize a single application is probably not a great idea. It passes on the (economic, not clock) cost to other developers.

4. Radically changing the filesystem to optimize a single application is also probably not a great idea. You are liable to break third party tools, windows tools, etc. that will corrupt the filesystem if they are allowed to run.

In practice:

Much of the stuff that belongs in the different zones resides in a single file. You usually store resources like icons within your executable file.

1. If you do decide to change the API, so that applications have to store their icons separately from their executable, and that sort of thing, many existing applications will not do this. You cannot just break all these applications, or you will get "Windows 8 is broken" from users, so you will need to keep the old 'deprecated' functions. There is absolutely no benefit to the developer of using the 'new and improved' interface, since from their perspective their is no improvement, so they will continue to use the deprecated functions. Even if it does start getting used, it will take 5+ years for the majority of applications to start using it. By that time, NTFS may be replaced, hard drives will have mostly become SSD, etc.

2. If you change the filesystem, say to add predefined tags, so that the filesystem knows which zone a file belongs in, the same argument holds. Also, you are liable to make the filesystem unreadable by earlier versions of Windows, third party tools, etc. You will also need to ensure that the OS is backwards compatible with the old filesystem: Samba will take a while to incorporate this as best it can, but if you don't do this you will break any number of file servers. People will not say "Oh. Samba is broken. It doesn't follow the new NTFS filesystem properly." They will say "Windows 8 is fscked. Don't let it near a corporate network."

3. If you avoid making these two changes, apart from the theoretical problem, you still need to face the problem of embedded resources. Should you split the file into two zones? This will slow down copying and writing the file. Or should you cache a copy of the resource in the correct zone. Caching is a useful tool, but apart from the space cost, it runs the danger of becoming dirty, eg. when your application crashes. Usually you can avoid this by cleaning the cache when your application starts. We are talking about an operating system, though. "When the application starts" means when the OS boots up. The cache would then actually serve to slow down the boot sequence. If you don't do this, you *will* get a dirty cache when you have a power failure, etc. But let's put that aside. The system will need to look in your home directory to see what's on the desktop and what's in your start menu. It will then need to see what resources the applications have, presumably by reading them. Then it will go to the cache to load the resource, rather than just getting it from the file.

I cannot see any obvious benefits here. What you can do, and this is what many other OSes do by default, is put the system and applications in one partition and the data in another. You can do this yourself. Just mount a data partition to the home directory. Obviously it's not a total solution because you probably have a bunch of applications sitting in your desktop or in your "Documents" folder within your home directory, but it may go some way towards your "zone" system.

No more disk map and other features that never came

stewartjazmin — Wed, 05 Aug 2009 11:32:23 GMT

One thing I was expecting from XP and forward is that I could get a better disk map, extended, showing the files or group of files on each "cluster" or "square" at that zoom level. That tells me where each file is.

Instead of that, and even when they say they've listened -"In Windows Vista, we had removed all of the UI that would provide detailed defragmentation status. We received feedback that you didn’t like this decision, so we listened, evaluated the various tradeoffs, and have built a new GUI for defrag!"

Say, a new one, with no info! (even not XP disk map)

I remember (where you took your defrag from) Norton, getting more and more intelligent and user configurable.

Automatic and user-defined features it supported.

Placing files according filter conditions at the beggining/end/middle of the drive.

This meant I could move the fixed size pagefile to the end, the boot files to the beggining (at that time there were no software updates), and the rest in the middle of the disk.

A complete disk map

A set of other feature I don't recall right now. But that was the Norton Utilities, before it was Symantec.

SSD Natural Fragmentation

stewartjazmin — Wed, 05 Aug 2009 11:40:55 GMT

SSDs have internal "wear-balancing" software that will intend to use ALL the cells in the drive the same amount of times. That means, you modify an existing file, and it will save the new blocks into a different place of a chip or in a different chip intentionally. No matter what the file system thinks where the file is really located.

So, even if you defrag them, or use a tool to image the drive and restore it (some of them restore the files in a contiguous fashion) it will fragment big time internally.