The release candidate of the Application Compatibility Toolkit is here, and it's good stuff. I've had to resort to sniffing around with a debugger to find many issues with this one, and it's still improving for the final release!

I am particularly impressed by what the team has put together for compatibility evaluators. For those of you who are unfamiliar with the tool, you can deploy a data collection package (DCP) that does more than just collect inventory - it actively shims system functions looking for known issues that could trigger compatibility issues after a migration to Windows Vista.

One of these is the UAC Compatibility Evaluator. It looks for a more limited subset of compatibility issues than Standard User Analyzer or LUA Buglight, because it is designed to keep the machine performing well even in a wide deployment to real users. However, one thing to keep in mind is that the UAC CE does not detect issues for applications that are not run elevated - it only detects these issues when the process is elevated and able to succeed. (I expect this may change for v.next.)

Now, for the vast majority of you, this probably isn't a big deal. Most organizations will perform their compatibility testing BEFORE they upgrade to a new operating system, for fairly obvious reasons! However, one great use of the evaluators that I have been recommending is to install it in your test lab while you are going through testing. So, while users or testers are running through their tests, you not only gather their feedback on the behavior, but you have additional technical data that perhaps may showcase issues that aren't a big deal in the lab, but may be a big deal when deployed to a lot more people. If you choose to take this approach, and you want to gather UAC data, then you will want to either consider one of the more diagnostic tools or else run the application elevated (but don't forget test runs with it non-elevated!).

I have run into a few scenarios where people want to be able to block access to Windows Explorer so that they can do something such as update the system in a machine that is otherwise publicly facing. One possibility is to create a desktop all your own.

The underlying architecture of Windows allows for something that may provide for this. Every instance of the operating system contains a collection of Sessions. Services run in Session 0, and interactive users run in Sessions 1, 2, 3, etc. (This is on Windows Vista - on Windows XP and earlier, the first interactive login shared Session 0 with services.) Each session contains a collection of Window Stations. Only one of these, WinSta0, is given access to display output, keyboard, and mouse. (Consequently, I haven't come up with any use in anything I have developed for the ability to create more.) Each Window Station contains a collection of Desktops.

You can already see multiple desktops just by using Windows. When you get to the login screen, that is a desktop. When your screen saver activates (assuming you are using a secure secreen saver), that has its own desktop. When you are prompted with a UAC dialog in Windows Vista, by default that has its own desktop. And you can create more. You can use the CreateDesktop API to create a new one, and then the SetThreadDesktop and SwitchDesktop APIs to switch to it. Here is a very simple example:

#include <windows.h>

int APIENTRY WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPSTR lpCmdLine, int nShowCmd) {
  HDESK hdeskOriginalThread = GetThreadDesktop(GetCurrentThreadId());
  HDESK hdeskOriginalInput = OpenInputDesktop(0, FALSE, DESKTOP_SWITCHDESKTOP);
  HDESK hdeskNewDesktop = CreateDesktop(TEXT("PrivateDesktop"), NULL, NULL, 0, GENERIC_ALL, NULL);
  SetThreadDesktop(hdeskNewDesktop);
  SwitchDesktop(hdeskNewDesktop);
  MessageBox(NULL, TEXT("MessageBox on private desktop"), TEXT("Private Desktop"), MB_OK);
  SwitchDesktop(hdeskOriginalInput);
  SetThreadDesktop(hdeskOriginalThread);
  CloseDesktop(hdeskNewDesktop);
  return 0;
}

This may immediately give you some ideas about kiosk applications. However, the desktop window manager (DWM) only runs on the primary desktop, so you won't be able to use Glass on any additional desktops you create. (Incidentally, that's also why UAC prompts are not rendered using glass.) So, if that's a consideration, then you may want to think of other approaches. But for some edge case scenarios, it's nice to know that you have this option available.

I ran into an interesting problem this week trying to get a legacy application running on Windows Vista. If you ran the installer, it would fail, claiming that it was unable to register a required component. So, I grabbed the ocx it was trying to install to have a look. Taking a peek by attaching a debugger to regsvr32, I could see that the component itself was just calling MFC42!AfxOleRegisterTypeLib, so it wasn't trying to do anything strange inside of DllRegisterServer. So, what was the problem?

Well, you can take a peek at the source for MFC, and see that it just calls in to OLEAUT32!RegisterTypeLib, so I started by setting a breakpoint there. It was reading through the type library, added registry values for the typelib itself, but then choked when adding the interface registry entries. Specifically, it was failing at OLEAUT32!CTypeLib2::IsValidHinfoDef. It turns out that we changed the format for type libraries quite some time ago, and we stopped supporting the old format as of Windows XP (but it was working fine on Windows 2000). This ocx was using the old format, so it couldn't register. How can you tell if your type library is doing the same thing? Set a breakpoint on OLEAUT32!CTypeLib2::IsValidHinfoDef and keep an eye on the EAX register. If you see it change from 0 to an HRESULT value (specifically 0x80004005), then it's basically telling you that the type library is not valid.

Of course, understanding the issue is only half of the battle - if you are doing serious application compatibity work, you aren't done until you have either fixed it or proven it unfixable. I hadn't proven it unfixable yet, so I had to start thinking about how to fix it. The type library is embedded as a resource, so we could theoretically replace it, but first we have to generate a new one with the updated format.

So, I just walked over to a Windows 2000 machine and used OLEVIEW to crack the binary and extract the typelib. Conveniently, it generates IDL from the type library it finds. I then copied that IDL, stuck it on a USB drive, and brought it over to my Windows Vista box. After updating the IDL to be compliant with current formats, I ran it through MIDL.exe to generate a new type library. I then opened up the ocx in Visual Studio, which gives you a nice view of the resources in a file. I added the new tlb, deleted the old one, renumbered the tlb resource to 1, and saved. Voila - same binary, new (and valid) type library. Running regsvr32 this time succeeds - hooray! Now we just need to verify that the component we registered doesn't have other compatibility issues!

I ran into an interesting situation with a customer's application the other day. You would try to install the application, and it would prompt for a reboot. OK, so it couldn't grab that file handle yet, I'll go ahead and let it. After a reboot, the installation would continue, and then prompt for a reboot again. And this cycle would pretty much continue indefinitely. What was going on?

Since the installation was an MSI, getting to the bottom of this one was a bit easier. Just pop open the Orca SDK tool, and take a peek. In an MSI database, you have a FeatureComponent table. This links to the Component table in the MSI. So, one by one, you can just see which components it is trying to install. And, since it was failing, you can always use the trusty EventLog, which conveniently provides the ComponentId which it was unable to write.

And here is where I found the culprit. But first, some background...

In Windows 2000 and Windows XP, we wanted to protect critical Windows sytem files. However, if we locked them down, then a lot of existing software would break, since the installers would try to install Windows components. You know, just in case they weren't there. So we had to be more clever about it. What we ended up doing was implementing a detection mechanism. We would detect that you modified one of the Windows system files, allow you to do this, and then come along later and just put the original copy back.

This was imperfect for a couple of reasons.

First, the copy was not immediate. I know the pain of this one first hand. I had an application that believed that not only did it need to install critical Windows system files, but it needed to install pretty much all of them. No problem, the original ones were dropped back in place and everything was fine. Then, I went to uninstall the application. It assumed that, since it dropped those critical Windows system files, it would be polite enough to go ahead and remove them for me. Just to be thorough. The problem was, it prompted for a reboot immediately afterwards. And, inconveniently, before the system file checker had a chance to lay them back down for me. So, upon reboot, Windows was unable to start because those files were never replaced. Yuck.

The other reason is that you could always get around this. We implemented this by laying down two copies of the system files. One in the normal location, and the other in the dllcache directory. Which means you could circumvent the protection if you were smart about it. If you lay down your copy in the dllcache directory first, and then lay down in the original location, then the system file checker would replace the copy in the original location with the copy in dllcache, which would be your modified copy.

And that is exactly what this application was trying to do. It wanted to replace the keyboard handlers from Windows with its own, so it was laying them down in dllcache first, and then laying them down in the original location. This little trick used to work on Windows 2000 and XP.

It doesn't work so great any more.

Because our previous method of approaching the problem was imperfect, we came up with a better one. We now ACL critical Windows system files to only allow the TrustedInstaller user to modify them. Not even System can modify these files, and certainly not Administrator (elevated or not). This is called Windows Resource Protection. How do we keep existing applications from breaking, since this is the reason we didn't implement the old technique in the first place? We lie to them. We detect if you are an installation program (using the same set of heuristics that we use to automatically mark setup applications to elevate), and if you are, then we accept your request to modify a protected system resource, we return success, but we never actually do anything to that file. If you are not detected as a setup program, then we don't lie - we return the access denied message. (Of course, you can apply the WRPMitigation shim using Compatibility Administrator if you need that behavior elsewhere, and we disable this check once you manifest your application with a Windows Vista manifest.)

As a result, we dont have dllcache any more. So, this system was trying to drop a new file into dllcache, it wasn't able to because that folder isn't there any more, so it figured that a reboot would help. And every single reboot, it wasn't there, so it figured yet another reboot might help next time. And on it went...