Why does it always seem to happen on Friday afternoons? Hardly ever does someone ring up on a Tuesday morning and say "my web site is down, please fix it!". Always Fridays. So there I was, having said goodbye to colleagues from India and Germany who had just set off for home after visiting the UK for 2/1 weeks respectively and about to settle down an an afternoon remotely debugging a customer's JScript application that was leaking memory, when Tess rang me and pointed out the severity 'A' escalated case that had just popped into our queue from our colleague Stefano.  We were debating whether to toss a coin for who would take the case when I realised that even with her superior debugging skills the 20 minutes before she had to leave the office might not be enough to resolve this customer's problem. So the task was mine...

 

The problem

The customer had an ASP.NET web site that most of the time, though not always, was giving the following error on every page:

Server Error in '/' Application.
Illegal characters in path.
Description: An unhandled exception occurred during compilation using the CodeDomProvider 'Microsoft.CSharp.CSharpCodeProvider'. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.ArgumentException: Illegal characters in path.

When I say most of the time, I mean sometimes the error was not occurring at all but once the error was occurring that was generally it until he performed and IISRESET. The customer was finding it hard to pinpoint exactly when the problem started. It had been going on for some time and they had also tried reverting the system to a previous configuration without success.

What was nagging at me was that this error was vaguely familiar and I eventually figured out that it was because I'd come across this once before when helping out a customer who had written a CGI application in .NET 2.0. After solving that particular case I wrote a KB article about it:

Error message when a CGI program that is written by using the .NET Framework 2 makes Web service calls: "System.ArgumentException: Illegal characters in path"

 

Data gathering

You can't do better than see a problem with your own eyes so as soon as I managed to get hold of the customer I set up an "EasyAssist" session to the customer's machine. EasyAssist is a tool we use within Microsoft support for remote viewing of and interaction with a customer's desktop. It is based on Microsoft Live Meeting but has a more limited feature set targeted at the support role.  All it needs is the ability to access the web through a browser and a small piece of client software that gets downloaded to the customer's machine for the duration of the support session. As most customer's I deal with can usually access the affected server or server from their desktop through a Windows Remote Desktop session I usually EasyAssist to the customer's workstation and go from there.

My colleague Stefano had already gathered some memory dumps of the aspnet_wp.exe process soon after these exceptions had been thrown.  These had given us the callstack of where the exception was occurring but unless you have a dump taken at the exact point of the exception you don't have access to things like what objects were lying around in the stack at the time. The stack looked like this:

 

Thread 9
ESP EIP
0x0549f3ac 0x77e55e02 [FRAME: HelperMethodFrame]
0x0549f3d8 0x799b5775 [DEFAULT] Void System.Security.Permissions.FileIOPermission.HasIllegalCharacters(SZArray String)
0x0549f3ec 0x799b5419 [DEFAULT] [hasThis] Void System.Security.Permissions.FileIOPermission.AddPathList(ValueClass System.Security.Permissions.FileIOPermissionAccess,SZArray String,Boolean,Boolean,Boolean)
0x0549f410 0x799b532c [DEFAULT] [hasThis] Void System.Security.Permissions.FileIOPermission..ctor(ValueClass System.Security.Permissions.FileIOPermissionAccess,SZArray String,Boolean,Boolean)
0x0549f428 0x799e96ee [DEFAULT] String System.IO.Directory.GetCurrentDirectory()
0x0549f440 0x0331d725 [DEFAULT] [hasThis] Void System.CodeDom.Compiler.CodeCompiler.Compile(Class System.CodeDom.Compiler.CompilerParameters,String,String,String,ByRef String,ByRef I4,String)
0x0549f478 0x0331cbd5 [DEFAULT] [hasThis] Class System.CodeDom.Compiler.CompilerResults System.CodeDom.Compiler.CodeCompiler.FromFileBatch(Class System.CodeDom.Compiler.CompilerParameters,SZArray String)
0x0549f4d4 0x03316695 [DEFAULT] [hasThis] Class System.CodeDom.Compiler.CompilerResults System.CodeDom.Compiler.CodeCompiler.FromDomBatch(Class System.CodeDom.Compiler.CompilerParameters,SZArray Class System.CodeDom.CodeCompileUnit)
0x0549f518 0x033164d3 [DEFAULT] [hasThis] Class System.CodeDom.Compiler.CompilerResults System.CodeDom.Compiler.CodeCompiler.FromDom(Class System.CodeDom.Compiler.CompilerParameters,Class System.CodeDom.CodeCompileUnit)
0x0549f534 0x033163c6 [DEFAULT] [hasThis] Class System.CodeDom.Compiler.CompilerResults System.CodeDom.Compiler.CodeCompiler.System.CodeDom.Compiler.ICodeCompiler.CompileAssemblyFromDom(Class System.CodeDom.Compiler.CompilerParameters,Class System.CodeDom.CodeCompileUnit)
0x0549f564 0x050acb2c [DEFAULT] [hasThis] Class System.Type System.Web.Compilation.BaseCompiler.GetCompiledType()
0x0549f594 0x050ac915 [DEFAULT] [hasThis] Class System.Type System.Web.UI.PageParser.CompileIntoType()

 

Interesting. We're throwing this System.ArgumentException whilst querying the CurrentDirectory. What on earth could be wrong with it?

What we really needed was a dump at the point of the exception so we could try and find out what was wrong with the current directory. Stefano had already tried this, giving the customer an ADPlus config file to try to generate a full memory dump when this exception occurred. However the expected dump was not produced and the issue was getting urgent.

Once on the box, I tried to refine the config file with my own first of all changing it to ignore all exceptions completed, then enabling only CLR exceptions (SEH code 0xe0434f4d) and adding a custom action to check if the exception type was ArgumentException and generating a full memory dump if it was.

But for some reason the system wasn't having it. Even with this minimally invasive approach things were grinding to a halt. With time pressing on a not a lot to lose I went for the live debug approach. In this way, thanks to the problem being readily reproducible at this time, I soon got the needed memory dump.

 

Analysis

In the dump at the top of the managed stack we had:

 

0:009> !clrstack

Thread 9
ESP EIP
0x0549f3ac 0x77e55e02 [FRAME: HelperMethodFrame]
0x0549f3d8 0x799b5775 [DEFAULT] Void System.Security.Permissions.FileIOPermission.HasIllegalCharacters(SZArray String)
0x0549f3ec 0x799b5419 [DEFAULT] [hasThis] Void System.Security.Permissions.FileIOPermission.AddPathList(ValueClass System.Security.Permissions.FileIOPermissionAccess,SZArray String,Boolean,Boolean,Boolean)
0x0549f410 0x799b532c [DEFAULT] [hasThis] Void System.Security.Permissions.FileIOPermission..ctor(ValueClass System.Security.Permissions.FileIOPermissionAccess,SZArray String,Boolean,Boolean)
0x0549f428 0x799e96ee [DEFAULT] String System.IO.Directory.GetCurrentDirectory()

 

and some of the objects at the top of the stack looked useful too:

0:009> !dso
Thread 9
ESP/REG Object Name
0x549f2d8 0x16596d4 System.ArgumentException
0x549f2f0 0x16596d4 System.ArgumentException
0x549f304 0x16596d4 System.ArgumentException
..

0x549f410 0x16596a4 System.Security.Permissions.FileIOPermission
0x549f414 0x16595e4 System.String \\?\C:\WEBSites\WWWMyApp\scripts

 

If you've ever looked glancingly at the output of !dso (short for !dumpstackobjects, a command in the SOS debugger extension) you may have thought, "hang on, why are there 3 ArgumentExceptions?". Well of course there aren't really; it is simply that lying around in the stack memory for this thread there are three pointers to the same ArgumentException object in the managed heap.

A frustrating thing when you are new to debugging of production .NET applications is that you don't get output that shows you the values of the various parameters passed from one function to the next. This is because in release builds the JIT compiler does not generate enough information to allow the debugger to extract this information; it is often optimised into registers rather than being in the stack memory.

But despair not! You soon get used to reading the output of !dso from top to bottom in conjunction with reading the !clrstack output and inferring what things are associated with what functions. And if you want a little more certainty than that then use the frame pointer as your guide. In the above !clrstack output, the ESP pointer for the GetCurrentDirectory frame is 0x0549f428. And in the !dso output at address 0x549f414 we have an interesting looking string "\\?\C:\WEBSites\WWWMyApp\scripts" .  So it easy to infer that this string is a local variable within that function and almost certainly the current directory of the aspnet_wp.exe process at the time of the exception.

 

Solution

First of all, what is illegal about this path? Well, nothing, if you are a Unicode Win32 API. As you can read in Naming a File on MSDN, certain Unicode Win32 file handling APIs allow a path to be prefixed with \\?\ which allow paths to be up to 32,000 characters in length among other things. It also tells the operating system to not canonicalize the path by interpreting things such as .. to mean 'go to the parent directory'. Unfortunately not all parts of the System.IO namespace in .NET have yet caught up with this reality and still consider ? in a path to be illegal.

So that is why we were getting the exception but how was the current directory of ASPNET_WP.EXE ending up as some path under the root of the customer's websites rather than being the .NET Framework root directory as it should have been?

I had a chat with the customer to get a broader picture of what else was on his server and one thing that came up that was interesting was when he mentioned an ISAPI component they used. This it turned out had been migrated originally from some old CGI code and now ran as a low isolation component loaded in the INETINFO.EXE process. And when the customer showed me where on the system this ISAPI DLL was located, sure enough it was in the same 'scripts' directory that ASPNET_WP.EXE thought was it's current directory.

At this point a theory came to mind. What if this ISAPI DLL was messing with the current directory of the INETINFO.EXE process and as a result ASPNET_WP.EXE (a child process of INETINFO.EXE) was ending up with the same current directory? That would explain it.

So first I needed to verify for sure that the current directory really was what I had suspected it was, given the string in the stack.

The current directory of a process, like so many other things about a process, is tracked within the Process Environment Block or PEB so we can find it in the dump:

 

0:009> !peb
PEB at 7ffd9000
..

440000 4216692d Feb 18 22:16:13 2005 \\?\C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\aspnet_wp.exe

..

0:009> dt 7ffd9000 ntdll!_PEB -r2

..
+0x010 ProcessParameters : 0x00020000 _RTL_USER_PROCESS_PARAMETERS

..

+0x024 CurrentDirectory : _CURDIR

..

+0x000 DosPath : _UNICODE_STRING \\?\C:\WEBSites\WWWMyApp\scripts\

So yes, the current directory of ASPNET_WP.EXE was indeed abnormal.

Next thing was to get a hang dump of INETINFO.EXE (cscript.exe adplus.vbs -hang -pn inetinfo.exe -quiet).

Looking at the current directory in the PEB confirmed that INETINFO.EXE also had this unexpected current directory.

My suspicion was that the ISAPI DLL, especially given its history as a CGI application may be calling SetCurrentDirectory but could I prove it? A live debug was not really an option as this was a live production web site. It was after normal business hours on a Friday so the customer would not have been able to get holder of the component vendor until Monday.

A quick way to check is to look at the import table of the DLL. If the import table refences SetCurrentDirectory then it means it has code that calls it. (It doesn't prove it called it but I'd but a fair bit of money on it.)

So how do we check that in a dump?

WinDBG comes with a number of extensions, one of which (ntsdexts.dll or exts.dll depending on which version of Windows you are debugging) contains a handy !dlls command. This lets you dump out details of the PE header and in there we can find the relative address of the Import Address Table Directory:

 

0x0012f3c0: \\?\C:\WEBSites\WWWMyApp\scripts\myapp.dll
Base 0x02d80000 EntryPoint 0x02d83891 Size 0x00009000
02d800f0 Opt Hdr
5000 [ 199] address [size] of Export Directory
7000 [ 50] address [size] of Import Directory
0 [ 0] address [size] of Resource Directory
0 [ 0] address [size] of Exception Directory
0 [ 0] address [size] of Security Directory
8000 [ 290] address [size] of Base Relocation Directory
0 [ 0] address [size] of Debug Directory
0 [ 0] address [size] of Description Directory
0 [ 0] address [size] of Special Directory
0 [ 0] address [size] of Thread Storage Directory
0 [ 0] address [size] of Load Configuration Directory
0 [ 0] address [size] of Bound Import Directory
7184 [ 134] address [size] of Import Address Table Directory

We can now use DDS to dump out the import table:

 

0:000> dds 02d80000 + 7184
02d87184 77e4e9fd kernel32!lstrcatA
02d87188 77e68990 kernel32!lstrcmpiA
02d8718c 77e6ef38 kernel32!lstrlenA
02d87190 77e65fa0 kernel32!GetModuleFileNameA
02d87194 77e57722 kernel32!lstrcpynA
02d87198 7c82f89b ntdll!RtlFreeHeap
02d8719c 77e70188 kernel32!GetPrivateProfileSectionA
02d871a0 7c82f9fd ntdll!RtlAllocateHeap
02d871a4 77e68537 kernel32!GetProcessHeap
02d871a8 7c82f4c3 ntdll!RtlGetLastWin32Error
02d871ac 77e6ba30 kernel32!WaitForSingleObject
02d871b0 77e4616d kernel32!DeleteFileA
02d871b4 77e75aed kernel32!GetTempFileNameA
02d871b8 77e622c6 kernel32!lstrcpyA
02d871bc 77e7061a kernel32!WritePrivateProfileStringA
02d871c0 77e56b5f kernel32!GetTimeZoneInformation
02d871c4 77e6c1ea kernel32!CloseHandle
02d871c8 77e6f42e kernel32!WriteFile
02d871cc 77e41840 kernel32!ReadFile
02d871d0 77e6f238 kernel32!SetFilePointer
02d871d4 77e41a38 kernel32!CreateFileA
02d871d8 77e4247d kernel32!CreateProcessA
02d871dc 77e56f84 kernel32!SetCurrentDirectoryA
02d871e0 77e453e4 kernel32!GetTempPathA

That's it. In IIS5.1 and IIS6 we prepend the \\?\ modifier when loading ISAPI DLLs or launching CGI applications as a security measure (to reduce the risk of path canonicalization attacks). The ISAPI DLL was almost certainly making calls to SetCurrentDirectory passing in the directory it was loaded from as a the new current directory for the process. This is a very inappropriate thing to do as a DLL, since a DLL is only a guest in the process space and should not be doing things that modify process wide attributes like the current directory. This is a bit like the situation that Tess had to deal with recently where a component was calling SetProcessAffinityMask and screwing up everyone else's day, including the .NET garbage collector.

 

A final irony

I was writing this blog post using my current blogging tool of choice, Windows Live Writer, which happens to be written in .NET. Everytime I entered that funny path into the blog text and hit the space bar of course Live Writer tried to be helpful and assumed I wanted this converting to a hyperlink. All well and good I think, I'll just go round and remove those unwanted hyperlinks later.  I was getting to the point where I'd typed enough that I wouldn't want to lose it if the tool crashed on me (it is a beta afterall) when the auto-save-drafts feature kicked in. And what should pop up on my screen?

My heart sank. My blog posting flashed before my eyes as I struggled to think of ways to save all that text I'd just typed (attach WinDBG, search virtual address space for words I knew were in the posting, saving that chunk of memory to disk, etc, etc.).

Anyway, fortunately Live Writer seems to have a good exception handler and was able to cope with falling victim to this shortcoming of the System.IO namespace. The tool didn't crash and I managed to remove the hyperlink before the next draft was saved!

 

Bye for now

 

Doug