Yesterday, a friend asked for some assistance debugging an issue with Windows Installer. It was one of those situations where you've looked at the problem so closely, so many times, that you just needed another set of eyes. (They're close to shipping, so they've been doing nothing but look at bugs, probably for a while.)

Some background:

What they're doing is a rather interesting use of Windows Installer. They're using it to drop a setup.exe and then run it. So, they have a custom action of type 0xC02, which is a deferred custom action, promoted to not use impersonation, which runs as an exe. This custom action spawns another version of itself (with different arguments), and then the process exits so the installer can complete. The second version of this custom action then wants to uninstall the msi, as well as launch the setup.exe. So, the msi is just a way to package up the setup.exe.

And, with UAC, it was breaking.

Now, with UAC issues, you typically think of access denied. But we didn't expect that at all here. We had promoted the custom action, so it should be running as local system. How could local system be receiving an access denied error? Perhaps we weren't being promoted for some reason?

So, I took a peek with Process Explorer. The processes had the privileges I expected, Local System. Here's the chain of process creation and token inheritance:

So, I could pretty much rule out a permission denied issue - the custom action was being promoted as I expected it to be.

My next step was to turn on MSI logging and see what was going on. And, as it turned out, it wasn't getting access denied (which I already expected), but instead was getting a 1605 error:

MSI (s) (5C:F4) [20:42:05:761]: MainEngineThread is returning 1605

Now, 1605 is ERROR_UNKNOWN_PRODUCT. Basically, it means you're trying to uninstall something that isn't there. This seemed odd, since the time stamp indicated that the previous installation had completed, and the rules of Windows Installer tell us that we couldn't possibly have two concurrent instances of the installer running anyway. So, what happened?

Take a look at the colors above. I originally invoked msiexec.exe from a protected admin account. I then attempted to uninstall that product from the Local System account. Aha!

You see, this msi hadn't set the ALLUSERS property. That meant that we were running a per-user install. When Local System tried to uninstall, it couldn't, because it had never been installed for *that* user, it had been installed for the Protected Admin user!

When I added the ALLUSERS property to the property table, and then set the value to 1, then everything worked as expected. And this little repackaged installer was working as expected.

Close one more bug. And UAC gets a free pass on this one.

It looks like Tech Ed IT Forum in Barcelona is sold out! I'll be presenting a couple of sessions there on application compatibility there, including the first public presentation of my shims deep-dive. (I actually did this one at an internal conference a few months back, and received some encouraging feedback on it.)

Here is my tentative calendar of talks:

CLI319 - Windows Vista Application Compatibility Tools and Resources
Date: 12 November 2007 Start: 16:00 Finish: 17:15 Room: Room 115

CLI01-IS - Application Compatibility Testing: Notes from the Field
Date: 12 November 2007 Start: 17:45 Finish: 19:00 Room: Room 130

CLI404 - Mitigating Application Compatibility Issues in Windows Vista using Shims
Date: 13 November 2007 Start: 17:00 Finish: 18:15 Room: Room 115

If you're going to be attending the conference, I would love to hear your feedback on all of the content we've been making available around application compatibility. My colleague Peg will be there as well (I don't think she blogs, so I didn't link to her) and she would also love your feedback. She has been trying to prepare all of the content that you would need to get started and maximize your chances of success. At the same time, I've been trying to get a lot more of the deep dive content out there. If you're an enterprise customer who has learned all of the tools and solved all of the problems except the last few hard ones, I hope I can give you the information you need to fix them. If you're a partner who is trying to help multiple customers resolve issues, and you want to know everything that we know, I want to share it with you.

Lately, I've been focusing a lot more on shims. This is just the feedback I've heard from the customers I've worked with and the events at which I have spoken: "I can find the issue, but I don't know how to match issues with a mitigation, because there isn't enough documentation on the mitigations!" I hear you! But what else do you want to hear to give you the knowledge and expertise you need? Comment, come and chat - make sure we know what you still don't know but need to know. I hope to see some of you there!

When I work with folks who are either new to debugging, or else want to sharpen up their chops (perhaps they have gotten rusty by not having to do so for a long time), one of the things I tell them to do is to watch out for things that don't do what you expect, and then figure out why. I mean, does everything on your computer doing exactly what you would want it to do, all of the time?

For example, I was working with a friend who doesn't have as many chances to debug things as he would like. Concomitantly, he was struggling with MS07-052, which (for the first couple of days) would cycle in an infinite loop for anybody who didn't install Crystal Reports as part of Visual Studio. It would install, succeed, and then detect that it needed to install again. He was about to ping a discussion alias about it. So I looked at him, and I said, "why don't you debug it?"

It's that kind of mindset that gets you practice debugging. Now, in this case, the cause was known, so he wasn't going to be helping anybody out. But in many cases, the cause may not be understood (you may be holding a repro that is really hard to get) - take advantage of that opportunity. And, even if you are doing some redundant debugging, if you learn something new and keep your skills sharp, would you begrudge yourself the time? If not, then perhaps...

One of the blogs I love to follow is Mark Russinovich's blog. He takes this exact approach over and over, outlining exactly how he uses tools to follow that investigation. And, obviously, learning how to better use the tools is one great takeaway.

But I think there's more to it than sharpening your skills with tools. Rather, it's a view into the mindset of somebody who is so natively curious. He sees a problem. It annoys him. So he figures out why it's happening, so he can get it to stop. By doing that alone, you can improve your skills exponentially, and Mark is kind enough to share his techniques for how to do that investigation.

One thing that I noticed that caught my attention even more than the great example of when to start debugging and how to do it was his awesome example of when to stop. In Mark's latest entry, The Case of the Frozen Clock Gadget, he proceeds through the investigation, and in the end determines the API that is causing a memory leak, after which he ... (drum roll) ... stops debugging it and files the bug. He doesn't dig into the implementation itself to see where the memory leak is happening. He never gets to the exact cause. But, what he has done is find somebody to assign the bug to, and then he moved on. There will be a single person who owns that API, and he can hand that person the symptoms and a repro. He can now go on to bigger and better things, and that person can fix up the code.

That, my friends, is a hard skill to learn. I mean, look at what I just said. You have to be curious enough to delve deeply into a problem at the first sign of trouble. You have to be creative enough to dig in and figure out what is going on. But then, before you see the actual culprit, you stop. You have to be able to turn off your curiosity, and know that your job is just to find an owner for the bug, and move on. (Don't worry, there are still plenty more bugs out there.) For those who are so natively curious as to get into the nitty-gritty of debugging, this can be a hard lesson to learn.

I see this all of the time working with enterprise customers who are testing application compatibility for a migration to Windows Vista, and one important step is knowing when to stop debugging. The answer, of course, can vary:

• If the application is one that you will never run without full vendor support, stop debugging before you start. If you're not going to ever bother fixing it, then knowing what's wrong won't help you, will it? This one is a hard one to learn. Many times, I have said, "I can debug this, figure out if I can shim it, and then test it with the shims applied, but if I do, are you going to run an unsupported, shimmed application? If not, then we're just wasting our time."
• If the application vendor isn't being responsive, or is playing the blame game ("It's the fault of xxx software that I depend on - if they'd just fix their bugs, then my software would run great - go talk to them.") then some debugging time can help you end the circle of blame and keep you talking to the people who can really help you. Remember,you're their customer, and if you come in saying, "I know it's you, you're doing this, this is illegal as documented here, and I need you to fix your code please" then you can circumvent the blame game and start getting some results. Note that, once you determine the culprit and can prove it, your debugging is done. Remember, you aren't going to change the code - the vendor is. Whichever one it turns out to be.
• If you don't need support, and are willing to shim the third party application, then you need to debug until you can figure out which shim you should use. Once you know this, and you shim it and the tests pass, then you are done. If you prove that it can't be fixed using a shim, and you need to code changed, then you are done.
• If the application is developed in-house, then you need to debug it until you know which person owns the component that is breaking. Then, assign the bug to them and let them finish debugging - your work is done.

Believe it or not, knowing when to stop is one of the hardest part of debugging applications. You need to encourage your innate curiosity to determine just enough, but then be able to turn it off. It's simply the only way to get everything done. And, believe me, by pointing somebody in the right direction, your help can be beyond invaluable. Happy debugging!

The last time around, I suggested that you avoid using the acredir.dll shims - RedirectRegistry and RedirectFiles. As alternatives, I recommended VirtualRegistry and CorrectFilePaths.

Of course, I have already gone into some details on how to use VirtualRegistry to achieve that, but I haven't gone in to any details on CorrectFilePaths yet. And, unfortunately, the documentation isn't much help as of the 5.0.1 release (as I said - stay tuned to the documentation for future releases, as this is something we are working on). In fact, here is all that the documentation tells you:

"Corrects file paths that changed between Windows 95 or Windows 98 and Windows XP Professional. This compatibility fix works by converting the file paths to the correct location for Windows XP Professional in the APIs. For example, a Windows 95 path of C:\Windows\Write.exe is converted to C:\Windows\System32\Write.exe."

While this is true, it is not comprehensive (and is, therefore, not as helpful as you probably want it to be). You see, CorrectFilePaths does apply a number of default corrections, but the real power is your ability to add additional corrections. (You can also configure the shim to not apply the default corrections, using the -b switch.The b stands for base, or bare, or whatever helps you to remember it.)

To add additional paths to correct, you once again have to go into the parameters dialog box. For this, once you have selected (and checked off) the CorrectFilePaths shim, you then need to click the Parameters button. Then, at the command line, you can enter the pairs of file paths you would like to have corrected, using the syntax "old path;new path".

The quotes I have used are not accidental. You see, part of mastering the use of shims on Windows is learning the intricacies of the command lines, and the person who developers each shim is responsible for parsing that command line. In this instance, the command line happens to be space delimited. So, if your paths contain spaces, you'll want to quote them. However, you don't want to quote each path (which is probably what I would have guessed), so "old path";"new path" is wrong. Rather, you want to quote the entire pair, and repeat this for each individual pair if you want to correct more than one path.

Here is an example command line from a redirection that I applied, to illustrate:

“C:\Windows\Downloaded Program Files\StockViewer.ini;%userappdata%\StockViewer.ini”

Now, one thing you may have noticed is that I have used what appears to be an environment variable. Let's talk about that.

There are a couple of reasons why we anticipated having to use CorrectFilePaths: hard coded file paths, and permissions issues. We either want to redirect a hard coded path, or we want to redirect from a location where a standard user won't have permissions to write to one where they do. (An aside about that: you may be asking why we would need to do this, given that File and Registry Virtualization in Windows Vista will redirect from per-machine locations to per-user locations. Well, we only correct the directories Program Files, ProgramData, and Windows. There are still plenty of applications out there who feel that they are "too important" to be lumped in with the other applications you have installed under Program Files, so they install in the root of the drive. We don't redirect for arbitrary directories, just for these three, so applications like this are going to need a big of a nudge from you.)

Clearly, with hard coded paths being one thing we're trying to fix, we don't want you to have to include hard coded paths in your command line, so we added the ability to get at known locations without that. But, since all you get is a command line, we didn't want to limit you just to environment variables, so we give you a slight variation that lets you get at the value from either an API or a known folder. Specifically, you can choose to use the following variables:

%windir% = value from GetWindowsDirectoryW
%systemroot% = value from GetWindowsDirectoryW
%systemdrive% = first two characters from GetWindowsDirectoryW
%systemdir% = value from GetSystemDirectoryW
%username% = value from GetUserNameW
%programfiles% = value from ShGetFolderPathW(..., CSIDL_PROGRAM_FILES, ...)
%userstartmenu% = value from ShGetFolderPathW(..., CSIDL_STARTMENU, ...)
%allstartmenu% = value from ShGetFolderPathW(..., CSIDL_COMMON_STARTMENU, ...)
%userdesktop% = value from ShGetFolderPathW(..., CSIDL_DESKTOPDIRECTORY, ...)
%alldesktop% = value from ShGetFolderPathW(..., CSIDL_COMMON_DESKTOPDIRECTORY, ...)
%userfavorites% = value from ShGetFolderPathW(..., CSIDL_FAVORITES, ...)
%allfavorites% = value from ShGetFolderPathW(..., CSIDL_COMMON_FAVORITES, ...)
%userappdata% = value from ShGetFolderPathW(..., CSIDL_APPDATA, ...)
%allappdata% = value from ShGetFolderPathW(..., CSIDL_COMMON_APPDATA, ...)
%allusersprofile% = value from GetAllUsersProfileDirectoryW
%userprofile% = value from GetUserProfileDirectoryW
%appexedir% = value from GetModuleFileNameW

So, you can use CorrectFilePaths to redirect to one restricted or moved location to a better one, leveraging these variables to locate the correct spot on a particular installation.

There are some additional details on this shim and how it can be configured. Specifically, we cover a huge set of APIs, you can add some more, or turn some off. But this isn't something you typically have to do. So, instead, I figured I'd give you the stuff you're actually likely to need, and include the rest in the less narrative documentation.

As we continue our journey through shims provided by the Windows Shim Infrastructure, we have reached a point where I can no longer avoid a discussion of the shims contained in acredir.dll: RedirectFiles and RedirectRegistry.

Last time around, I pointed out how to redirect the registry using VirtualRegistry, which requires a command line argument which (for a very long time) was not publicly documented. Instead, a lot of people have been using RedirectRegistry, for two important and logical reasons. First, the name just sounds so tempting, since that's exactly what you likely want to do. Second, the act.chm actually does document these shims. So, I understand why people would rationally flock to these shims.

However, one of the internal questions that I address quite a bit: why can't I get xxx application to work with the RedirectFiles shim? Am I putting in the command line wrong? (and the command line follows)

So, I'll just say this. You shouldn't use these shims, because they're not designed to do what you think they're designed to do. Specifically, they target Internet Explorer.

Try it. From Compatibility Administrator, press the Query button on the toolbar. The second tab, fix properties, allows you to find programs fixed using a shim. Put one of these two shims in there, and then press the Find Now button. IE7 is the only thing that comes up. Now, try either VirtualRegistry or CorrectFilePaths. Lots and lots of things come up.

You see, IE7 leverages the Windows Shim Infrastructure to resolve an important compatibility issue: attempts by ActiveX controls to write to areas that Mandatory Integrity Control prevents them from writing to in LORIE, implementing an Internet Registry and Internet File System - a separate Virtual Store than that used by the operating system - and these two shims help make that happen.

So, these two shims are specifically designed and optimized around this one scenario, and were even separated off into their own DLL to help keep them separate. The problem is that we include the acredir.dll shims in Compatibility Administrator without this caveat. They have tempting names. We give them good documentation. So it's no surprise that people use them.

Instead of RedirectRegistry, I recommend using VirtualRegistry, which can redirect (among other things). Instead of RedirectFiles, I recommend using CorrectFilePaths (which I haven't talked about here yet, but I'm getting there).