Zipping up Unicode file names

Interesting - I've submitted this to WinZip.
Another question occurs to me.

Suppose John zip's up a file containing some eight-bit characters in his code page.

Then he emails the file to Mischa who has a different code page.

Mischa extracts the file.

What does he see? Are the eight-bit characters in the file name just whatever those eight-bit numbers happen to map to in his code page?

I wonder if there's any kind of ../.. vulnerability possible here.

../.. is something to watch out for if Unicode filenames are allowed, though. It's easy to imagine a malicious zip file containing a "filename" of "../../../../../../boot.ini" - where the .'s or "/"'s are obfuscated with overlong Unicode escapes.

Plus, if you used the short filenames, you'd not be able to expand them back to their originals after unzipping them, making it kind of useless anyway (it'd be better to manually rename to something in English I reckon).

Perhaps an extension to WinZip (or whatever) could be to encode the Unicode characters using utf-7 or maybe even using the same scheme as IDNs or something? That way, you'd still be able to unzip with another program, just not with the friendly filenames.

We noticed this problem when doing the packaging and deployment support for VS6.

The net result is that this is why _A_UTF8 was added to the fci/fdi headers for CABs; we just utf8 encoded the filenames in the CAB as it was already 8-bit clean. (I claim that the cab code owner added it at my request but maybe someone else beat me to the request...)

For ZIPs we had a similar issue but I believe that ZIPs for JARs were already specified as being either 7-bit clean or UTF-8 encoded already. I think we forced a fix in the Java Package Manager to deal with UTF-8 encoded class names but these still tended to have problems on Win9x since the underlying registry APIs couldn't deal with characters in key or value names not representable in the system's CP_ACP.

Actually NTFS-conformance problems are very common with Win32 software. Many do not understand that NTFS is Unicode (and then I haven't mentioned ADS).

The scariest part is that most file managers also have complete ignorance of Unicode. This can be funny when you have an English locale and you try to copy a file with e.g. cyrillic or just East-European characters in its name.

Another typical area of problems is with email clients.

I tried to compress these files with RAR and 7-Zip (http://www.rarsoft.com/ http://www.7-zip.org/)...
They work well, all files in both archives are extractable as original names. So it seems to be only zip format problem.

There is unfortunately some ambiguity within Zip files as to the character set that the filename is stored in. Whenever a file can be stored in the OEM character set (which is the character set originally used by MS-DOS), WinZip does so for compatibility with other Zip utilities. In this case, it marks the Zip file as made by MS-DOS, since the original Zip utilities were DOS utilities that used the OEM character set for filenames.

One thing to note is that on NT-based systems such as Windows NT, Windows 2000, and Windows XP, file and folder names are stored as Unicode (a multi-byte character set that includes characters from various ANSI and OEM code pages). Such files and folders can only be processed by Unicode-aware applications. WinZip is not a Unicode-aware application at this time. For this reason, WinZip can only display and process characters that exist in your current codepage. If your current code page is English 1252, for example, you will not be able to see or process files that have Chinese characters. On the other hand, if your current code page is set to Traditional Chinese - 950, you should see and be able to process files with Chinese characters but you may not be able to see or process files and folders whose names contain Greek characters.

We are looking into how to handle this better in a future version of WinZip.

Please let me know if you have any further questions.

--Chuck Campbell, WinZip Technical Support

It works just fine in Mac OS X. The Mac uses UTF-8, which would be the ideal choice for encoding filenames in the existing zip format.

When I brought the zip file onto my Windows box, WinZip thought αβγδεζηθ.txt was αβγδεζηθ.txt. You can see the UTF-8 encoding in there, mapped to 1252.

Too bad Windows doesn't have a UTF-8 API. It already has a UTF-16 API. I wish I could use the ANSI APIs but tell the system to use UTF-8 as my current encoding scheme.

The Unicode exploit I was thinking of is if the "../" check is only placed against the raw string, and not the normalized form.

For example if the string is stored in UTF-8, a [c0][af] byte sequence will not trigger the path validator. On the other hand it may very well be normalized during the extraction routine - boom, a vulnerability.

Such vulnerabilities are not new to the Zip world - see http://secunia.com/advisories/8781 for an example of path interpretation vulnerabilities (though admittedly this one is not Unicode-related.)

The ZIP file format specs are here:
http://www.pkware.com/company/standards/appnote/
A quote:
"The current Header ID mappings defined by PKWARE are:
...
0x0008 Reserved for future Unicode file name data (PFS)"

So, it seems something is there already. But someone should bite the bullet and use it. Probably the backward-compatibility is a problem, but sooner or later someone should do it :-)