Long Paths in .NET, Part 3 of 3 [Kim Hamilton]

BCL Team Blog : Long Paths in .NET, Part 2 of 3: Long Path Workarounds [Kim Hamilton]

Tue, 10 Jun 2008 18:07:25 GMT

PingBack from http://blogs.msdn.com/bclteam/archive/2007/03/26/long-paths-in-net-part-2-of-3-long-path-workarounds-kim-hamilton.aspx

re: Long Paths in .NET, Part 3 of 3 [Kim Hamilton]

Mike — Tue, 10 Jun 2008 20:21:36 GMT

Hmm, using short file names sounds more like a hack than a real solution.

For one thing short file names exist in Window just for the sake of compatibility. Using what is basically an obsolete feature of Windows to "fix" .NET which is supposed to be "the future" sounds backwards to me.

Second don't be so quick to assume that turning off short filename generation is uncommon. Generation of such names is known to slow things down so don't be too surprised if people turn it off. Even if they don't know how to do it some enterprising person may include this feature in an "optimization" tool and spread it across. This means that .NET developers won't be able to relay on this "fix" working on every computer.

Here's a case where people had to turn short name generation off because of performance issues: http://channel9.msdn.com/forums/TechOff/407930-SQL-Server-2008-CTP-FILESTREAM--NTFS-limitations/

If .NET framework is to fix this somehow I think it is better to simply allow \\?\ (even if this will need full trust to work). This will put .NET framework on par with Win32 and who knows, maybe one day Win32 will increase the limit and limit the need for \\?\.

re: Long Paths in .NET, Part 3 of 3 [Kim Hamilton]

Kim Hamilton — Tue, 10 Jun 2008 21:15:09 GMT

Before you start throwing around the word "hack" you might re-read and notice that I also proposed we support \\?\. :) However, we would not prepend this automatically behind the scenes and it would require some (tbd) heavier demand than FileIOPermissions.

The auto-shrinking case is what we could attempt behind the scenes to soften the impact for the "slightly above" cases, where users don't necessarily want to get into the world of \\?\.

re: Long Paths in .NET, Part 3 of 3 [Kim Hamilton]

Eric — Tue, 10 Jun 2008 23:27:22 GMT

Put me down as a strong "no" for this approach.

The fact that Windows allows users to turn off short file names should be reason enough. I might be wrong, but short file names were basically a kludge added for backwards-compatibility in Windows 95. It's even further moved into the "compatibility" realm when dealing with Windows NT based operating systems.

One thing I've been very happy about over the years is that short filenames are effectively "dead" - nothing uses them by choice any longer.

Adding it to .NET to support a fringe case (path lenghts > MAX_PATH) will effectively make it a "core case" - something that all applications will start using. Effectively, you will be reintroducing the "hell" (no offense) of short paths to the world once again.

I've seen existing bugs in .NET (can't remember which classes) where it "faked" short paths by truncating then concatenating ~1 to the end - regardless of what the underlying filesystem will do. You have no guaruntee before you touch the NTFS filesystem that it *really* will turn into ~1 in the Win95 days files with the same beginning would become ~2, ~3, etc. This is the deal breaker - you cannot guaruntee compatibility as long as this is true. If I have a directory called "Program Files" and another called "Program Data", your algorithm will specify the wrong one! You will not know if the directory is PROGRA~1 or PROGRA~2 before you make the API call unless you make other I/O requests to the NTFS / storage system.

As stated in your post - the BCL is effectively limited by what the Win32 API has done (inconsistent \\?\ support). Why is this any different than what Win32 API developers face? They have the same issues - some support long paths, some do not.

I'd rather see a "modern" solution that follows the recommended practice set out by the Operating System (Kernel) guys. That practice is to use \\?\, not introduce kludges that are only there for backwards compatibility with DOS.

"Shrinking" paths will be a *LOT* scarier (conceptually) for a developer than having to know which paths are supported in which context.

It still doesn't fully work around the MAX_PATH issue either - the developers who are asking for a workaround for MAX_PATH are envisioning \\?\, which is not the same as using short filenames to have "a bit more room". They want the full meal deal, not a faked version.

And if you're not satisfying the core audience who are requesting this feature, why would you introduce a "DOS" kludge into .NET? It just doesn't make sense.

re: Long Paths in .NET, Part 3 of 3 [Kim Hamilton]

Kim Hamilton — Tue, 10 Jun 2008 23:35:38 GMT

Hmm, these comments are strange given that I propose explicit use of \\?\ should be allowed. Maybe people are skimming and not noticing? In any case, to avoid having to repeatedly say "read the blog" I'll make some updates to ensure that's clear.

re: Long Paths in .NET, Part 3 of 3 [Kim Hamilton]

Eric — Tue, 10 Jun 2008 23:37:37 GMT

As an example to my previous post, try the following on Windows Vista:

* Open Command Prompt

* Change directory to C:\

* Type "dir /a /x"

Notice the short filenames displayed? On my machine, "ProgramData" is "PROGRA~2".

With the proposed algorithm, the file:

"C:\ProgramData\<really long path>\MyFile.txt"

Would be translated into:

"C:\PROGRA~1\<shrunk path components>\MyFile.txt"

However, you now have non-deterministic behavior. The file will actually be saved as:

"C:\Program Files\<really long path>\MyFile.txt", which is not what the developer intended.

That said, perhaps additional API calls will be used to perform the actual translation and "lookup" pre-existing files.

However, wouldn't the proposed behavior still be non-deterministic for files being created?

re: Long Paths in .NET, Part 3 of 3 [Kim Hamilton]

Lionel — Tue, 10 Jun 2008 23:40:05 GMT

The solution using short names looks messy, and IMHO it might look safer in the short term, but it has a lot of potential for creating compatibility nightmares in the long term. It makes path canonicalization even more complicated, it would have a cost in performances by requiring disk access or an in-memory cache for short names lookups, whether path are too long would depend on the length of usually hidden short names, and it will often end up in displayed paths and a poorer user experience.

I agree with Mike's description: it's a hack. Granted, it's an attractive one, which would work in the short term, but you will end up paying the "long paths tax" later, and it will be more expensive.

My advice would be to start supporting long path using \\?\ and optional canonicalization in .Net, and push for legacy applications to be upgraded (starting by Windows components and APIs). It will take somewhat longer, but you will end up with less clutter in your code base, and less nightmares with unintuitive behaviors in applications. Moreover, it would not be that bad: long paths are still uncommon, not a critical feature, and once the basics tools (especially Windows Explorer) support them, there will be a clear reason for third parties to start supporting them.

Do not forget, too, that not supporting long paths often mean static buffers, potential overflows truncations or error cases, and thus an area where security vulnerabilities could creep in.

re: Long Paths in .NET, Part 3 of 3 [Kim Hamilton]

Lionel — Tue, 10 Jun 2008 23:48:52 GMT

Kim,

Whether you allow \\?\ in parallel with path shrinking is mostly irrelevant. You would still be introducing more complex path handling rules, relying on short names, and add a lot of potential complexity in .Net's behaviour. The drawbacks are still there.

Moreover, if you give a simple, partial solution and a solid, harder to use one, most programmers will choose the first. The option to explicitly use of \\?\ would be mostly ignored. On the other hand, if you focus on using \\?\ behind the scene, you could try to create an easy to use, cleaner API, and try to hind the quirks of the underlying Win32 calls.

re: Long Paths in .NET, Part 3 of 3 [Kim Hamilton]

kimhamil — Wed, 11 Jun 2008 00:29:33 GMT

Note that I rearranged the post to highlight that \\?\ is indeed part of the solution and that auto-shrinking helps "make it work" for opening existing files when \\?\ is not used.

I have to agree that my mention of \\?\ support was rather buried before -- basically it was the closing sentence of the "solution" part. I guess I can't expect everyone to pore over every sentence. Hope it's clearer now. :)

re: Long Paths in .NET, Part 3 of 3 [Kim Hamilton]

Max Christian — Wed, 11 Jun 2008 01:44:09 GMT

Please be aware that 8.3 filename generation has to be disabled for any application that creates and reads files rapidly, and has more than a few thousand files per directory. Otherwise the performance is terrible -- there is a KB about it which I can't find right now.

So disabling 8.3 generation is much more common than you might think.

re: Long Paths in .NET, Part 3 of 3 [Kim Hamilton]

Thomas Krause — Wed, 11 Jun 2008 01:55:33 GMT

IMO you should prepend \\?\ automatically for all .NET APIs that support it (but still enforcing the other path restrictions of course). .NET APIs that wrap Win32 functions without \\?\ support, should use the proposed automatic path shrinking if the path is too long, so that they'll be also able to handle long paths whenever possible.

Of course you will still have problems with other applications that don't support long file paths, but these applications would not work with path shrinking either, if they don't shrink paths themself.

Also while the proposed solution will only work if short file name generation is not deactivated, this solution would still be able to work in most (or many) cases.

What do you think?

re: Long Paths in .NET, Part 3 of 3 [Kim Hamilton]

Dean Harding — Wed, 11 Jun 2008 03:16:06 GMT

Eric, I don't think you understand the algorithm that is being proposed. It wouldn't be blindly truncating and appending "~1" to the file name. It would use the *correct* short name for a given long name. I can't believe how anybody could imagine it working any other way.

As to the original question, given that \\?\ will be supported anyway, I'm not sure what the benefit of path shrinking would be. It would work some times, but not all times, and so you'd still need to work around the times it doesn't work with \\?\.

I think path shrinking should be limited only to those cases that won't be fixable by supporting \\?\ -- namely LoadLibrary and that sort of thing.

re: Long Paths in .NET, Part 3 of 3 [Kim Hamilton]

orcmid — Wed, 11 Jun 2008 05:42:42 GMT

First, I think this is a great discussion to also reflect on the MSDN Technical Interoperability Forum, because I suspect there are many more things that haven't been thought about, and attracting more eyeballs could be valuable.

With regard to short names, I think people don't get that the short names are always there (right?) for any non-short name of a directory or a file. Nicht wahr? You want to see some, run edit.com (yes, that one) from a console window. Then notice what the console shows as the current directory path afterwards. (So would we be breaking cmd.exe with this stuff?)

The interoperability issues I am thinking of have to do with (1) shared directories that have these monster name snakes in them, (2) software that reads and writes related formats, such as ISO CD-ROMs and DVDs. Oh, yes, ahem, flash drives and that Windows Home Server on my workgroup LAN. There are APIs that have hard-coded limits for passing filenames back and forth too (there are ones in ODMA that I am probably too familiar with).

Then there are all those web file uploaders and downloaders, etc., etc. Probably one more way to break Media Player in an inscrutible way, etc.

I hope the wider conversation around this happened internally (on that internal list you mentioned in one of these posts). Is there some place where people triage things that will cost more in support incidents than the problem they solve?

Hmm, yes, I think this goes onto the Technical Interoperability forum and maybe some wider readership, though not sure what that is.

re: Long Paths in .NET, Part 3 of 3 [Kim Hamilton]

Dean Harding — Wed, 11 Jun 2008 06:19:30 GMT

orcmid: Notice the comments above about how short name generation can be turned off. While it's true that turning off short name generation won't delete the short names that *already exist*, it obviously won't generate any new ones, so it'll only help you in some situations.

I believe the main problem with shrinking the paths is that it only works some of the time, and it's totally arbitrary whether it'll work on any given path (so your application might be OK on an operating system that has short name generation turned on, but if your customer has it turned off, it might not work for them... I imagine it's going to be very difficult to track those sorts of problems down -- better to fail consistently).

re: Long Paths in .NET, Part 3 of 3 [Kim Hamilton]

Yggy — Wed, 11 Jun 2008 06:58:34 GMT

As part of the long-term plan, have you considered deprecating strings-as-paths altogether and using the Uri class instead?

NTFS Streams?

David — Wed, 11 Jun 2008 13:52:37 GMT

If you're changing the filename parsing in the BCL, what about allowing names that specify alternate NTFS streams? i.e. c:\MyFile.Txt:extra_metadata

At the moment (well, last time I tried) it will get upset about the extra : character.

MATTER SIZES, aka Knowing when to unflip a[n inappropriately flipped] bozo bit

Sorting it all Out — Wed, 11 Jun 2008 21:28:59 GMT

The source of the main title is an inside joke I am probably not going to ever explain within the blog.

re: Long Paths in .NET, Part 3 of 3 [Kim Hamilton]

Peter — Wed, 11 Jun 2008 21:37:12 GMT

Firstly, just swapping one very large limit for another isn't really going to solve the problem. People would simply start using very, very long directory names (think of how long URLs get when you start adding on form data -- and a couple of guids -- and a session id -- and the date -- and, and, and).

Secondly, making the limit bigger imposes horrid problems on small devices. My kids have some "clock radios" that can play MP3 files (currently favorite: Oliver!). These have enough trouble with MAX_PATH names.

Thirdly, path names also have to fit into the user interface. The Explorer in Vista already has enormous troubles with the existing path sizes; the last thing it needs is to have more troubles.

Lastly, stop it, stop it, stop it. File paths are a SYSTEM thing to be solved by the SYSTEM groups. The last (deleted) thing I need to deal with is a zillion (deleted) "solutions" by a zillion (deleted) groups, all "solving" the problem in their own, incompatible way.

Suppose you DO come up with a "solution". How will my C++ app deal with it? Will I have to make a call from C++ into .Net just to call your lame API? How will I do it from my installer? Can I do it from the Java program that runs my builds?

re: Long Paths in .NET, Part 3 of 3 [Kim Hamilton]

orcmid — Wed, 11 Jun 2008 23:41:13 GMT

Urk about short-name generation being disabled.

So, I agree, this is a SYSTEM issue and in particular a matter with regard to the file system.

I guess I don't know what the use case is that has exceeding MAX_PATH be locally useful while remaining globally safe.

More fodder: Zip files and directory-simulating Zip access,

and then there's the part: protocol and OPC

um, the file: URI,

oh, and who gets to figure out all the threat-modeling of this combined with NTFS streams, hiding root kits, etc.

and what about using long-path injection as a form of buffer-overrun exploit, although I suspect that would be hard (assuming the Win32 APIs never change and neither does the MAX_PATH constant, for obvious down-level protection reasons).

Just tossing things around here. The bigger problem with this conversation is that the inside one is completely separated from the outside one and who knows what well-trodden ground we are revisiting. So I think I will shut up now. I hate write-only feedback channels.

re: Long Paths in .NET, Part 3 of 3 [Kim Hamilton]

Jon Grant — Thu, 12 Jun 2008 11:29:40 GMT

I think you're overanalysing this WAY too much.

The fundamental problem is that .NET says that "?" is invalid in a path. Remove that restriction. That's all you need to do.

"Oh but developers might create paths with invalid characters and spaces at the end!"

SO WHAT! Stop holding our hand. .NET is not VBScript for dummies.

Right now, we are forced to re-implement the whole System.IO namespace purely to get over this shortsighted and rediculous piece of validation. In fact, I don't see wht the framework has to validate paths at all: just pass it down to the API and let it throw an error if it's not happy with it.

Stop nannying us. We're not idiots. Stop making assumptions on our behalf.

re: Long Paths in .NET, Part 3 of 3 [Kim Hamilton]

akraus1 — Thu, 12 Jun 2008 15:56:41 GMT

Hi,

I tried you sample code from your blog but I cannot get it to work on XP SP2. Is the \\?\ mechanism not supported for really long file names?

StringBuilder sb = new StringBuilder("C:\\SubDir\\SubDir");

for(int i=0;i<60;i++)

sb.Append("FileName");

sb.Append(".txt");

LongWriter.TestCreateAndWrite(sb.ToString());

The Directory does already exist. I get as return value: The filename, directory name, or volume label syntax is incorrect.

Is there a known limitation with the \\?\ approach?

Yours,

Alois Kraus

re: Long Paths in .NET, Part 3 of 3 [Kim Hamilton]

akraus1 — Thu, 12 Jun 2008 16:42:00 GMT

Ahh found the issue. The fsutil command did lead the way:

C:\>fsutil fsinfo volumeinfo c:\

Volume Name : System

Volume Serial Number : 0xf43ee3c8

Max Component Length : 255

File System Name : NTFS

Supports Case-sensitive filenames

Preserves Case of filenames

...

It seems that the directory name or file name itself cannot be longer than 255 characters. The 32K limit apply for the full path but the path parts itself cannot be longer than 255 characters still.

Yours,

Alois Kraus