If NTFS is a robust journaling file system, why do you have to be careful when using it with a USB thumb drive?

re: If NTFS is a robust journaling file system, why do you have to be careful when using it with a USB thumb drive?

Matt — Tue, 08 Jan 2013 09:06:38 GMT

@f0dder: "You didn't read my entire post? Let me quote the relevant part: "Sparse Files were obviously counterproductive to the goal of avoiding fragmentation :)"

Well sparse files are the only way to achieve what you want - and the fragmentation of sparse files across disk is a fairly small problem - ranges are automatically collected and nearly full sparse files are automatically converted to non-sparse files by background tasks in Windows. Sparse files are ordered collections of "ranges" of blocks where blocks outside of those ranges are zero. All other files have a "valid data" pointer where the data before that value holds non-garbage data on disk.

Consequently if you want to write the last 4KB of a 1GB file without writing the other blocks to disk you have to either use sparse files, or suffer the large write of zeroes back to disk.

re: If NTFS is a robust journaling file system, why do you have to be careful when using it with a USB thumb drive?

Roman — Tue, 08 Jan 2013 03:04:02 GMT

> But the point of discussion is "If I yank a thumb drive before delayed writes are complete, why doesn't journaling fix the problem?"

Oh, I see. This makes sense. May I humbly suggest that the title of this entry is somewhat misleading then? It _sounds_ like you intend to discuss why you can't just yank an NTFS formatted USB drive: note the part where the title seems to assert that "you have to be careful when using [NTFS] with a USB thumb drive" - but you go on to clarify that this only applies if delayed writes are permitted.

re: If NTFS is a robust journaling file system, why do you have to be careful when using it with a USB thumb drive?

Nawak — Mon, 07 Jan 2013 21:42:52 GMT

>> Ultimately, on graceful close, the reserved allocation will be truncated to the current file length, even if there was a reservation.

> [Um, the discussion is about what happens when there is not a graceful close

@Raymond: Indeed, but alegr1's remark can apply to the fsck done during the drive remount. Since the last operation didn't finish, the allocated space could be reclaimed. Also, reading back I think that alegr1's remark applied to Matt's second comment about some (apparently small) files' allocation staying very large after a browser crash. In that case, the OS would do the graceful close (from the FS POV) on process exit, wouldn't it?

@Matt:

1) Reserve being part of quota: well, ok, but could as well be! I think including would still be fair (since reserved space isn't available to other users), so another decision I'll have to contest! ;-). Note that I don't advocate having a "reserve" marginally different from the "size" after a close (or remount). This would only be a hint for opened files.

2) Implementation detail? Let's call it optional functionality! If the reservation promise isn't kept then it's only a performance degradation for the offending FS. So I think every FS could return OK without doing anything. Reserve space qualify as a hint and isn't therefore contractual so no API would be broken in your case. If you want to talk about implementation details, being able to fill up a file with zeroes at no cost when resizing it, *that* is an implementation detail! And implementing this could be way harder for another FS than answering OK to a hint it didn't take.

3) You are right. So that's always A->B->C with A being optional. So if you implement a service that allocates and set the reserve, there's no extra cost at all, since you still have to at least update the commit at each write (commit+filesize in my solution, but maybe the informations aren't in the same physical places?)

4) I already acknowledged that. The benefit was that the corruption was more visible.

@Matt and others: Yes, none of what I propose would magically uncorrupt a file from a drive that has been removed before the end of the operation. The initial point was to make it more visible that something went wrong, even if I admit that not everyone knows what size a particular file should be... Wait! No, the start of all this was my surprise of seeing Raymond say that having the final size at the end of the auto-repair was the correct behaviour. I now know that it is indeed correct considering what CopyFile asked. Hilarity ensued when I asked for another service! ;-)

@f0dder:

- I agree that when an FS/API is already designed, coded and tested, that kind of service would have to gain a lot of points to justify the work. Still doesn't make the requested behaviour "incorrect" or wrong to examine.

- for your second "2)": there wouldn't be more metadata updates than there already is. As Matt noted, you still have to change the commit size at each write. If my uninformed opinion is correct, you could update the size in the same write.

re: If NTFS is a robust journaling file system, why do you have to be careful when using it with a USB thumb drive?

f0dder — Mon, 07 Jan 2013 17:59:47 GMT

@Matt: "Yes. Sparse files."

You didn't read my entire post? Let me quote the relevant part: "Sparse Files were obviously counterproductive to the goal of avoiding fragmentation :)"

re: If NTFS is a robust journaling file system, why do you have to be careful when using it with a USB thumb drive?

Roman — Mon, 07 Jan 2013 17:18:42 GMT

Raymond, I think the confusion arose because your first sentence says you need to do X to enable formatting as NTFS, while later on you talk about the write-behind cache. Frankly, I'm still confused, even after your clarification :)

On my Win7, I was able to format a USB hard drive as NTFS without making any promises or agreeing to any warning dialogs. I assume the write caching is disabled, based on what you say about that being the default. Therefore, I should be safe to yank it and expect the just-copied *data* to survive too (not just filesystem metadata), right?

[If the rules changed in Windows 7, nobody told me about it. (Not that I expected them to.) But the point of discussion is "If I yank a thumb drive before delayed writes are complete, why doesn't journaling fix the problem?" The rules around delayed writes may have changed, but that doesn't affect the discussion. -Raymond]

re: If NTFS is a robust journaling file system, why do you have to be careful when using it with a USB thumb drive?

JamesJohnston — Mon, 07 Jan 2013 15:56:10 GMT

Raymond: "Write-behind caching is disabled by default on removable drives. You get into this mess only if you override the default."

xpclient: "I checked that same drive on XP/Vista/7/8 and XP blocks NTFS if it's set for Quick Removal and allows NTFS if set for Performance but Vista and later OSes allow formatting it as NTFS irrespective of whether it's set for Quick Removal or Performance."

There seems to be an underlying story here which I've somehow missed. Did Windows Vista introduce support for NTFS and Quick Removal? Did prior versions of Windows not support that configuration?

re: If NTFS is a robust journaling file system, why do you have to be careful when using it with a USB thumb drive?

Matt — Mon, 07 Jan 2013 08:52:29 GMT

@fodder: "is there a way to pre-allocating a file size and then writing to an arbitrary location in it, without (NTFS) having to zerofill from the former last-valid location to the current random offset?"

Yes. Sparse files.

re: If NTFS is a robust journaling file system, why do you have to be careful when using it with a USB thumb drive?

f0dder — Mon, 07 Jan 2013 01:52:40 GMT

I was thinking that having an in-memory "eventual filesize hint" for NTFS could have been a nice thing - something that wouldn't be used in the on-disk structures, simply a best-effort thing for the driver to take into consideration while the operation is still running. If the process crashes, file size would be trimmed to the (journalled as normally) FS metadata updated by the last write.

But after pondering a bit:

1) starts at -1000, is probably hard to do right, and doesn't bring much to the table.

2) doesn't solve the "file /contents/ might be corrupt" problem.

2) while (likely) reducing fragmentation, you still get "a zillion" metadata updates, so the copy will be slower than really pre-allocating the output file size.

Bonus question: is there a way to pre-allocating a file size and then writing to an arbitrary location in it, without (NTFS) having to zerofill from the former last-valid location to the current random offset? Last time I played around with that was back on XP, and SetFileValidData() seemed a bit dangerous and privilege-hungry for general purpose. Sparse Files were obviously counterproductive to the goal of avoiding fragmentation :)

re: If NTFS is a robust journaling file system, why do you have to be careful when using it with a USB thumb drive?

@xpclient — Sat, 05 Jan 2013 20:34:55 GMT

>>> "Do you notice a tone of complaint in my comment?".

Yes. Almost everything you say on Rayomond's blog is phrased in a critical way. Rather than "Bug or feature?" you could have phrased it less agressively, e.g. "why is that?" or "am I missing something?", rather than just assuming that things not working as you expect is because the feature is broken.

re: If NTFS is a robust journaling file system, why do you have to be careful when using it with a USB thumb drive?

alegr1 — Sat, 05 Jan 2013 19:57:10 GMT

FAT32 (artificial) volume size limit is 32 GB. Over than that, exFAT should be recommended. Actually, it makes sense to use exFAT for smaller drives, unless you plan to use the drive at systems where exFAT is not inbox, or not installed as update.

>how can a user be sure the CopyFile succeeded?

1. Make sure CopyFile issues FlushFileBuffers for removeable volume.

2. For a removable volume, limit amount of write-back to 1-2 seconds worth of data, and have a short write-back idle delay. This is for applications that don't use CopyFile.