Understanding NTFS Compression

Sparse Files vs Compressed Files with VMs

Anon — Mon, 07 Jul 2008 18:29:52 GMT

So you say that compressed files 'hurt' VHD performance significantly. Does this also follow true for NTFS sparse files?

I could believe the implementation for sparse files is lighter (RLE) and therefore quicker to return, but I suspect there's a performance hit caused by fragmentation when you have to expand the file? In which case is the performance penalty from the decompression or mostly from the file fragmentation (which could be corrected)?

[A sparse VHD is an interesting scenario and I would like to test that when I have a few free cycles.

In the meanwhile, here's what I would expect to find. Each random access to the physical device takes between 3-10ms (disregarding hardware cache). Because of this, performance hinges on how many random access reads & writes are necessary to/from the device. For example, hunting through the volume looking for 1 thousand 4KB fragments will be much more costly than a single contiguous 4MB read. The storage stack may break up the 4MB contiguous read into max read units (depending on hardware & storage driver requirements), but it is still going to be much faster than the 1,000 individual reads.

If you step through a sparse file adding a few bytes to every other compression unit, you could quickly find yourself with a million fragments (and a very messy fragmented file). The host has very little control over how the guest OS allocates & frees disk space on the VHD, so I suspect that you would initially have good performance, but the VHD access would become increasingly slow over time.

Regards,
Dennis]

re: Understanding NTFS Compression

law — Sat, 30 Aug 2008 21:55:25 GMT

So suppose I am writing a file to disk (or modify data from a "CU") -- at what point does NT try to recompress the the data to try to save space? Each time the data would be written from a "CU"-size multiple from memory to the physical disk?

re: Understanding NTFS Compression

Reiner Saddey — Sun, 08 Mar 2009 12:49:23 GMT

I just want to thank you for your in-depth explanation of how NTFS compression is bound to create fragmentation.

I'm using Ntbackup to create backup sets within NTFS compressed folders and just wondered if and how fragmentation might be avoided.

Having read your article, I now know that short of running Defrag there's no way to avoid fragments. They should thus be considered a feature resulting from the space left by shrinking CUs.

Still I suspect that it might be possible for the compression to create contiguous file allocations in the first place provided those files are written sequentially. In other words, if the creation of a sparse file would proceed completely sequentially and if the compression would produce CUs in order form sequential application writes at EOF, contiguous allocation should be feasible without running Defrag.

Again, thanks to your team and keep up your great work,

Reiner

[Hi Reiner, DNA molecules describe the type, shape, and arrangement of cells within an organism. Mapping pairs are similar, but they describe the type, shape, and arrangement of clusters within a stream. Normally when you defragment an uncompressed file, the defrag utility rearranges the many fragments into one (or more) larger contiguous regions. When the allocations become contiguous, we can calculate a less complex DNA chain for the stream. Copy your file into a compressed folder and now your file has a new type of DNA with additional requirements. It would be possible to relocate the allocated portions of the CU’s into a logically contiguous space, but the number of DNA chains would remain the same. We must have one VCN->LCN mapping for each compressed allocation plus a VCN placeholder for any sparse regions. The VCN placeholders are required for alignment purposes, so they must be present regardless of how the allocations line up on disk. The end result is that the CU’s would be packed closer together, but the number of mapping pairs required to describe the CU’s would be the same. I hope this helps! .]