Chris Jackson's Semantic Consonance : ACT 5.0

What is Coming in ACT 5.5, and Should You Wait for It?

Chris Jackson — Tue, 17 Feb 2009 23:16:48 GMT

With the release of the Windows 7 beta, there has been a lot of speculation about an accompanying version of the Application Compatibility Toolkit. Because the release of a version of ACT so frequently accompanies the release of an OS (or a significant service pack, such as Windows XP SP1), we’ve generated this perception in the community that you have to have the version of ACT that is matched up with that particular version of Windows. (We see this a lot with people who are just now deploying Windows XP SP2 – they start looking around for older versions of ACT, which are no longer available.)

ACT 5.5 is scheduled to be ready by around April. (I’ve also done a piece for TechNet Magazine with my friend Chris Corio that goes deep into the internals of ACT which should be out at around the same time – stay tuned for that.) If you plan to start getting ready for a Windows 7 deployment, do you need to wait for it?

I don’t think you do.

But, rather than me just throwing out an opinion, let me walk through the process of an application compatibility project, point out what you could do today with ACT 5.0.3 (the currently available version), and then point out what you would get in addition if you were to wait.

Collect

What you can do today with ACT 5.0

You need an inventory of what software you have in order to drive your project. ACT 5.0 will give you an excellent inventory of what you have, so you can get started today figuring out what you have in your ecosystem. In fact, I recommend it, unless…

When you should wait for ACT 5.5

There is a compelling new feature in ACT 5.5 that may be worth the wait: the ability to “tag” a DCP package. If you have the ability to target deployment of your DCPs, and you’d like to have the applications in your inventory come back reflecting where they came from, then this may be worth the wait.

ACT 5.5 will include new deprecations in the deprecation agent – specifically, the Windows Mail deprecations. If you see Outlook Express / Windows Mail displaying UI in your organization, then this may be important. But, to be honest, I don’t expect this to have a significant impact on many enterprises at all, so unless you’re particularly worried about this, it may not be worth waiting for. (Obviously it’s useful once it’s out, but do you really want to set yourself back a couple of months to get this if you’d rather get started today?)

Analyze

What you can do today with ACT 5.0

It kind of depends on how much it bugs you to see “Windows Vista” in the report category as to whether or not you’re comfortable using it.

The compatibility data will be from Windows Vista and not from Windows 7, but not only are we not breaking that many more things, it’s not as if that data was terribly good to start with (even with Windows Vista, going to windows.com/compatibility was your better option).

And, of course, if you dump your inventory into Excel to analyze, then 5.0 is fine.

When you should wait for ACT 5.5

ACT 5.5 will display Windows 7 as an option in the report listing, so that will satisfy your sense of aesthetics. In addition, because the list of operating systems is starting to get kind of big, and you’re likely to only be migrating to one at a time, you will be able to filter them and only display the one you intend to deploy. That simplifies the UI a bit.

If you are worried about the data you send up to the ACT web service, ACT 5.5 is going to add on some functionality to help. 5.0 would always send the unique Application ID up – but if you unselected it we wouldn’t send any of your evaluation data. ACT 5.5 won’t even send up the ID (as a result, you won’t be able to see any community data for this app) and the tool gives you direct visibility into the exact data that you are sending up. For folks who are concerned about this, you may want to wait for this feature before you sync.

ACT 5.5 will also be able to support syncing with the Windows Compatibility Center – so no more manually looking things up at windows.com/compatibility! My understanding is that this may not be fully ready to go when the product ships, but it will be able to support it once the service is ready to go. I think that’s going to be huge.

Note that this doesn’t mean you can’t collect your inventory now – I haven’t personally tested the database upgrade scenario, but I have been loading up my sample database with the log files from an ACT 5.0 DCP deployment, and have had no problems doing so. My guess is that we’ll elegantly handle a database upgrade from 5.0 as well (I just don’t want to promise something I haven’t tried myself).

Test and Mitigate

What you can do today with ACT 5.0

Standard User Analyzer

SUA is relatively unchanged from the ACT 5.0 version. One big change: you need AppVerifier 4.0+ on Windows 7, and only the ACT 5.5 version works with AppVerifier 4.0+. So, you’d just have to do your testing on Windows Vista or earlier.

Internet Explorer Compatibility Test Tool

IECTT supports IE8 today, so you don’t have to wait for anything to start investigating your web applications!

Setup Analysis Tool

Does anybody use the setup analysis tool? It remains, and it’s as uninteresting today as it was for Windows Vista. This could be the last time we see this tool.

Compatibility Administrator

Compatibility Administrator is relatively unchanged, and you can use it today on Windows 7 without hesitation.

When you should wait for ACT 5.5

Standard User Analyzer

If you want to test with SUA on Windows 7 itself, you’ll have to wait. But for getting started to get your apps ready for a more secure environment today, you’ll get the same functionality down-level.

Last I heard, LUA Buglight also has a version check (alas, in the driver, so you can’t shim it) that keeps it from running on Windows 7, but current builds work fine on Windows 7. Not sure when Aaron plans to release an update.

Summary

Have we filled all of the gaps with ACT 5.5? Certainly not. I’d love to do more to help people rationalize apps. I’d love to have a smooth integration story (heck, I’d take a rough one if I could just get anything). I’d like a better 64-bit story (that’s on deck). I’d like more focus on the actual process than the technology. There’s always something else I want. But there is some good stuff coming (I’ll never be satisfied anyway).

In conclusion, the ACT 5.5 story falls very much in line with the rest of the Windows 7 application compatibility story. You can get started today and not incur too much risk of a tool or feature coming along later that is killer. The tags for DCPs may be a killer feature (they’re certainly a highly requested one), but otherwise you can get started today, and the work you do with the tools you have will translate to results in the future.

If you get started with your inventory and your rationalization, then you can just expect more data to come along later to help you out. In the interim, you can at least be identifying the projects that require a lot of work, and get started on them now. It’s simply a lot smarter to get any fixes you may need (to play nicely with the new security posture begun with Windows Vista and continuing with Windows 7) by working them in to the natural development and deployment cycle of your software today, rather than it is to try to squeeze them all in at the same time only when you begin the deployment of an operating system later.

I hope this little review is helpful for formulating your tools strategy if a migration to Windows 7 is in your future…

Standard User Analyzer Refuses to Run with Application Verifier 4.0 (and Application Verifier 3.x is Gone!)

Chris Jackson — Wed, 04 Feb 2009 21:01:00 GMT

Updated March 16, 2009: Somebody updated these links with the 4.0 version (which kind of defeats the purpose of having these links so I’m not sure what they were thinking) but they’re back to the 3.x version now.

Hey, there’s a new version of Application Verifier in town, and guess what? Standard User Analyzer doesn’t like it. ACT 5.0.3 (the latest publicly available version) is hard coded to look for versions 3.2 through 3.5, so the brand new version 4.0 kind of leaves this tool in a lurch.

It’s also a mystery to me why the #1 web hit on live.com for Application Verifier leads to http://www.microsoft.com/downloads/details.aspx?FamilyID=bd02c19c-1250-433c-8c1b-2619bd93b3a2&DisplayLang=en, which is a, “We are sorry, the page you requested cannot be found” page, which helpfully has search results from the same search engine, which helpfully has a #1 web hit of the same unhelpful page. Seriously? Did we just not pay attention to the fact that we should have kept this page and forwarded, since people don’t want to have to guess the GUID for the new version? A search for Application Verifier 4.0 brings up the desired download as the #2 hit, so all is not lost (just most).

But the fact that you can eventually find this incredibly useful and important tool if you try hard enough still doesn’t help fans of Standard User Analyzer (which are many). So, until we release ACT 5.5 (which support Application Verifier 4.0), we posted the 3.x versions again in a super-secret hidden location so you can still download them from us instead of from a random source. Unfortunately, we didn’t include fwlinks in the product so we could just update these links and you wouldn’t have to read this here, an omission we have already fixed.

So, here’s where you can get the 3.x versions of application verifier:

http://msdl.microsoft.com/download/symbols/debuggers/Private/ApplicationVerifier.ia64.msi

http://msdl.microsoft.com/download/symbols/debuggers/Private/ApplicationVerifier.amd64.msi

http://msdl.microsoft.com/download/symbols/debuggers/Private/ApplicationVerifier.x86.msi

Note that you really shouldn’t use Application Verifier 3.x on Windows 7 – you’ll want to use Application Verifier 4.x on it, which means you’ll end up waiting for ACT 5.5 to use SUA on Windows 7.

Shimming Applications on Windows Vista 64-Bit

Chris Jackson — Mon, 01 Dec 2008 19:47:38 GMT

The same question came up two times in 26 minutes (on the same discussion list, no less), so I figured I’d answer it once here as that seems a reasonable indicator that others may have the same question.

What is the deal with shimming on Windows Vista 64-bit?

Well, it turns out it’s a bit of a mixed story, so let’s go through it.

First, Compatibility Administrator itself runs just fine on x64. It’s a 32-bit app, so it’s running on WOW64, but that’s OK. You can run the tool to create custom shim databases to run on either x86 or x64.

Now, let’s talk about the shim engine. It’s installed on x64, and it supports shimming both 32-bit and 64-bit binaries. So far, so good.

But once we start talking about the custom shim databases you can create today using Compatibility Administrator, the story is not complete today.

You see, the custom shim databases you create (whether you create them on x86 or x64) will only shim 32-bit applications. So, while the platform supports shimming 64-bit applications, the tools don’t give you the ability to do so.

So, if you have a native 64-bit application that isn’t working on Windows Vista, then you are unable to use shims today in order to mitigate the problems you discover.

I haven’t come across any problems with native code doing this, but I have come across problems with managed code. There just isn’t much native code that is compiled for 64-bit (and what is available tends to be fairly recent). Managed code, however, defaults to compiling for “any CPU” – which means that the JIT compiler will compile native 64-bit code for managed code when it’s running on 64-bit. So, you can fix it on x86, but not on x64, even though the same app runs on both.

Well, that’s not very cool. That just might discourage you from moving to x64 if you have a broken managed code application, eh?

Well, there are two solutions for that. Obviously you could recompile the application and change the compiler flags to target 32-bit only (but if you’re doing that, you may as well fix the bugs!). If you can’t recompile, however, you’re not stuck. Just use corflags.exe to set the 32BIT flag, and you’ll be on your way.

By the way, if you really want to get under the covers, you can. The underlying cause is that the XML we generate doesn’t have the (optional) OS_PLATFORM attribute. Try it yourself – if you look in Process Monitor, you’ll see that compatadmin creates an XML file in appdata\local\temp. Of course, it deletes this file when it’s done, so you’ll want to catch it before we’re done with it – a quick glance at the stack tells you that you’ve got msxml3.dll in the stack when you’re creating this, so if you bm on msxml3!* and start stepping until the call to CloseFile, you can go into the temp directory and pull out the XML to see for yourself.

CorrectFilePaths Has to Point to a Directory Which Exists

Chris Jackson — Wed, 01 Oct 2008 01:43:40 GMT

A question came up via comments. (I was going to say that it came up recently, but another glance reveals that it came up in, oh, June. I don’t think I can fairly call that recent…)

“…the fix seems only to work if the directory structure exists…”

This is true, and worth noting. If you point the fix to a directory which doesn’t exist, the shim won’t create the directory, it will just fail, in much the same way the application would fail if it were trying to create an application in a directory which didn’t exist (which, to Windows, appears to be exactly what happened). So, if you’re planning to shim an application and want to create a directory to store things, you should consider creating this directory at install time.

CompatAdmin How Do I Shim Thee? Let Me Count The Ways...

Chris Jackson — Wed, 10 Sep 2008 21:13:37 GMT

When you come across issues debugging applications, there are typically several ways to solve them. Today, I'm going to pick on our own stuff and throw a few different shims at it. Interestingly enough, what I'm going to be shimming up will be the tool I use to create shims: Compatibility Administrator. That's right - I'm going to shim up my shim tool.

What's the bug?

---------------------------
Compatibility Administrator
---------------------------
You do not have administrative rights. Some features might be disabled.
---------------------------
OK
---------------------------

Yeah - I don't enjoy warnings like that, and I certainly don't need them every time.

Solution #1: shim up CompatAdmin.exe with RunAsAdmin. That makes the MessageBox go away because it actually is an admin.

What else could we do? Well, we could figure out which functionality need admin rights. Sitting there using the product non-elevated, I counted two broken features. First, I can't right click and disable a shim. Second, I can't install a shim database.

I don't really need to disable things often, and it seemed to me that installing is something that could be factored out. In fact, a quick glance at Process Monitor tells me that installing a shim database already is factored out! CompatAdmin.exe just calls sdbinst.exe. Is it calling it using CreateProcess (which would require ElevateCreateProcess) or ShellExecute(Ex)? ShellExecute. Nice. So, in theory the think I need to do actually should work just fine, except the software is stopping me from doing it!

---------------------------
Compatibility Administrator
---------------------------
You need administrative rights to perform this operation.
---------------------------
OK
---------------------------

A big fat LUA Bug in our own stuff. It stops me from doing something that works because it doesn't believe it does. Nice work. Let's lie to it.

Solution #2: shim up CompatAdmin.exe with ForceAdminAccess. The dialogs go away, and I can still do what I actually want to do - right click install works just fine, and prompts me when I do that. I then end up running less code elevated, and being more secure. I can read existing databases without elevations. What a huge win! (Well, except not being able to turn shims off, which may be interesting from time to time, but for the vast majority of what I do, this is a great solution.)

Are we done yet, or is there another way to skin this problem?

Well, these are messageboxes. We can suppress them one last way.

Solution #3: shim up CompatAdmin.exe with IgnoreMessageBox (parameter You do not have administrative rights. Some features might be disabled.,Compatibility Administrator). We'll get that to stop annoying me! It still shows the MessageBox when I try to install a database (which is not helpful) but if I suppress that one also, then it just does nothing and says nothing, which is arguably not a very good outcome. But at least it's a touch less chatty. I'll chalk this up as the worst solution.

Personally, I've been using ForceAdminAccess lately. But I thought this was a fun app to show the options available, particularly since you need to shim CompatAdmin somehow to make it useful. (Yeah, I know - that's kind of ridiculous and we should fix it. The bug has been open for over a year - I just poked the team again.)

Announcing ACT 5.0.3 (a.k.a. ACT 5.0.5428.1080)

Chris Jackson — Wed, 03 Sep 2008 04:56:13 GMT

It seems like just yesterday I was posting about ACT 5.0.2 being released, but we just released ACT 5.0.3.

Now, I've had a couple of people confused about the version numbers we talk about, and what they actually see. For, rather unfortunately, you didn't see 5.0.2 anywhere in the last one, nor will you see 5.0.3 in the new one. Why don't we do you the honor of showing that in the help - about? As it turns out, we branched way back when at RTW, so the Main branch lived on, while the RTW branch was where we finished things up. As it turns out, the 5.0.1, 5.0.2, and 5.0.3 updates where all released off of the RTW branch, so 5.0.5428.x has remained the naming convention, and we have just kept incrementing the build number.

ACT 5.0.2 == 5.0.5428.1056

ACT 5.0.3 == 5.0.5428.1080

So, that being said, what's new in the 1080 release?

Update Compatibility Evaluator: The Update Compatibility Evaluator now supports Windows Vista and Windows Server 2008 (Windows 2000, Windows XP, and Windows Server 2003 were previously supported)
Windows Internet Explorer 8 Support: We now support Windows Internet Explorer 8. The Internet Explorer Compatibility Evaluator as well as the Internet Explorer Compatibility Test Tool are capable of identifying issues on websites and web applications when they are run on Windows Internet Explorer 8.

Major Bug Fixes:

SUA:

SUA no longer crashes when the Application Verifier log entry is badly formed
No longer Fails to create shim database entries if the product name contains special characters
The CorrectFilePaths shim applied by SUA used to only fix a single path - this has been resolved and it now is applied correctly

ACM:

The Application Compatibility Manager no longer requires elevation of user privileges to run on Windows Vista (HOORAY!!!)

It does require you to uninstall the old toolkit first, and then install the new one. The schema changed, so you'll have to upgrade your database also (warning if you've got apps or scripts crawling around in there). Hopefully it's helpful!

Getting (Some) Virtualization Data from the Internet Explorer Compatibility Test Tool

Chris Jackson — Mon, 11 Aug 2008 21:47:19 GMT

If you run the Internet Explorer Compatibility Evaluator on Windows Vista, you get back some data when an ActiveX control tries to write to a file that a standard user used to be able to write to. However, that data doesn't show up when you are using the Internet Explorer Compatibility Test Tool.

Why?

It's configured not to.

Why?

Um ... I don't know? (Perhaps because the quality of the output is qualitatively different than every other test in the tool?)

But I do know how to turn it back on.

Open up Notepad elevated, and then open up "C:\Program Files\Microsoft Application Compatibility Toolkit 5\Internet Explorer Compatibility Test Tool\IECEConfig.xml".

If you then search for the word EXCLUDE, you'll find this:

<EXCLUDE>VirtualizationAction:AcTraceIgnored;</EXCLUDE>

Just comment this out (or delete it) and we'll stop excluding data from the view. Now your tool is configured to stop hiding data that must be useful. Is it perfect? Nope - it doesn't collect it all. But at least it shows you what it did collect.

Why doesn't it collect it all? I don't know, but I can speculate. The Internet File System and Internet Registry are implemented on IE7 using shims (specifically, shims sitting in acredir.dll, such as RedirectFiles and RedirectRegistry). While most of IECE's output is based on the browser itself, the browser would have to predict what the shim infrastructure was doing. While you can turn on shim diagnostics and pull some of this out depending on the debug spew and log spew the shim writer included, that really complicates things. So, unlike rules where the browser explicitly knows "hey, I'd have to do this bad thing which the following code will specifically prevent, better log it" here we have API interception happening at a completely different layer where the browser code would have to either attempt to predict (could have false outcome), hook into the shim infrastructure, or parse ACLs and duplicate security logic.

But the things we are detecting, we really ought to show, wouldn't you agree?

Documentation Update for the Application Compatibility Toolkit

Chris Jackson — Wed, 18 Jun 2008 21:17:51 GMT

Last week, we updated the documentation for the Application Compatibility Toolkit. It's kind of hidden, though - if you go to the Application Compatibility Toolkit download page, you will find a new item in the list of files:

ACT50_Doc_Update_Installation_Instructions.zip (1.9 MB)

It's got a bit of a silly name (because it's not just instructions - it's the update also) and a silly package (or no package, really - you drag and drop the new chm into a protected Program Files folder, so you'll get UAC prompted), but it is goodness.

Liz Ross is the technical writer who creates the docs for ACT. She's got some other goodies as well, and this is the latest manifestation of the workflow we've put together. I'm busy translating shim docs from source code to geek, and she's busy translating them from geek to human.

Have a look. I'm particularly interested in getting shim documentation out faster, and hopefully updating without waiting for the next release of ACT can help you be more effective resolving application compatibility issues.

We'd love to hear your feedback on the updated docs!

Resolve Issues with Windows Resource Protection using the WRPMitigation Shim

Chris Jackson — Sat, 31 May 2008 00:57:22 GMT

Every now and again, I bump up against a setup application (it's almost always a setup application) that tries to drop older versions of protected operating system files. It's fairly easy to mitigate, but I thought I would go through some of the mechanics, and some of the places where the mitigation can break down.

Let's take a walk down memory lane...

Introduced in Windows 2000 and brought over to Windows ME ... I'll pause for everyone to recover from the shivers in your spine ... we offered a feature called System File Protection. This feature is designed to protect the integrity of the operating system. This feature had a couple of issues, which I talk about here: http://blogs.msdn.com/cjacks/archive/2007/04/20/windows-resource-protection-wrp-and-activex-control-installation-on-windows-vista.aspx.

So now, things are better, except of course the applications that break, which is why we didn't just modify the ACLs in the first place. So, one by one, we just start applying our WRP shims to applications that need it, and this got us back where we needed to be for application compatibility. We automatically apply this shim if we detect that you are a setup. We apply this shim to regsvr32.exe. We throw it everyplace that we think we're going to catch people trying to write to protected operating system files.

So, how does it work?

Well, first we always try to run the original API. If that fails, then we check to see if you are a WRP protected file. We do that for performance - if what you are trying to do already works, we don't need to fix you, nor do we need to run code to determine if we ought to fix you. We just let you go on doing your thing.

But if the operation didn't work, then we'll check to see if the file is WRP protected. If so, then we'll pretend that things worked.

How do we pretend?

Well, if you're trying to delete, we can just return success. Move? Success. Change attributes? Success.

But what if you're trying to write?

In that case, we don't get very far by returning success, because we need to return a handle that is valid or the application could AV. So, we just create a temp file, and return a handle to this temp file.

Right now, I'm running an application attempting to update kernel32.dll, and I find this in %temp%: WRP112.tmp.

That's where I'll be writing when I write. My application continues to work, and we resolve the issue.

But let's back up a minute - what happens if I am trying to drop my file in a non-standard location and register it from there? Here, we can run into issues. Clearly, the application is intentionally trying to circumvent protection mechanisms. They drop the dll somewhere other than the location of the protected dll (which succeeds) and then call regsvr32.dll. (Some people think they're really tricky - I once saw an app drop some version of shell32.dll from Windows 2000, but they called it shell32.ico, then called regsvr32 on it.)

Our check to see if it's a protected operating system file says "no" because it isn't owned by Trusted Installer, and we don't fix it up.

With people trying to be tricky like this, I usually end up modifying the package to remove this drop. But I suppose I could CorrectFilePaths to hit my protected location...

Modifying Directory ACLs with the OpenDirectoryAcl Shim

Chris Jackson — Wed, 28 May 2008 23:01:48 GMT

I received a request to talk about a particular shim. And yes, I received that request over a month ago, so ... sorry about that. Nonetheless, I've managed to scratch out a little niche of time to discuss the shim:

OpenDirectoryAcl

Yep, this does exactly what you would expect it does, modifying the ACL on a directory. I had not documented this shim yet because in a typical enterprise environment, it typically is not the best approach to resolving application compatibility issues.

But, that does not mean that it is never useful, so rather than just ignoring it, I will simply go through the steps I would normally go through to eliminate the need for this shim, and allow my eminently gifted readers to re-use or modify the logic as it suits them.

Can you redirect the file write?

If I can redirect the file write to a less privileged location, then I prefer to do that. By doing so, I don't have to loosen the security profile of the system. Even if I need to share the file, I generally prefer to take writes out of Program Files and instead set something up (ACL'd appropriately) in ProgramData, so I will sometimes redirect even if I am changing directory or file permissions.

Should the application run elevated?

If the purpose of the application is to modify protected resources, it may be a better choice to require elevation rather than to remove the protection on the resources.

Can I modify the ACLs when I repackage?

If I have decided that I need to loosen the ACL. First, I re-read Changing access control on folders vs. files and think again - am I really sure I need to modify the ACL?

Let's say that I have some compelling reason to change permissions on the file or directory. Since most enterprise software is deployed using management software, such as System Center Configuration Manager, rather than simply waiting for user's to self-launch an executable file, then I would approach that by repackaging or transforming the installer (frequently an MSI) rather than shimming a setup.exe. The LockPermissions table in an MSI allows me to set ACLs. (If you're writing the installer using the fabulous WIX tool, then you can use the Permission Element.)

There are simply almost no situations where I find users self-installing setup.exe files. However, in the consumer space, this is extremely common, which is why this shim works well in that scenario and we see this bubbling up as the 11th most commonly used shim in the system shim database.

How OpenDirectoryAcl works

Why do you need to target the setup? Because that's when we apply the ACLs. Specifically, we intercept calls to the CreateDirectory APIs, and if you create a directory that matches the command line, then we put it in the queue to modify that ACL. Then, when we either call the ExitWindowsEx API or exit the process, we apply the ACL changes that were matched. That means you are modifying ACLs on directories when you create them, so you have to target the setup.

So, I think you can see now why I have never once had to use this with an enterprise customer. But just because I haven't doesn't mean you won't.

Configuring OpenDirectoryAcl

If you've gotten this far, well, let's configure this shim. The command line syntax is as such:

-<options> <dir-or-file>|<sddl> [ -<options> <dir-or-file>|<sddl> ...]

The options you can specify are:

r - recursively match files
m - match the directory part to expand to the installation directory
M - match the directory / file parts to expand to the installation directory
p - ACL the parent directory as well
P - ACL the parent directory only
f - create an empty target file if it does not exist (do not combine with "d")
d - create an empty target directory if it does not exist (do not combine with "f")

You then specify the directory or file you want to modify the ACL for. (Yes, the name of the shim is deceiving, because you can specify a file rather than a directory.) If the file name has spaces, then make sure you enclose the file name in quotes, as spaces are a delimiter for the command line. Also, if you use environment variables.

Finally, you don't need to specify SDDL. By default, you'll receive D:(A;OICI;FA;;;BU)

However, you can specify SDDL if you want. after the directory or file path, simply include a pipe character ( | ) and then include the SDDL you'd like to add.

Ask me about this, or other shims, at TechEd 2008

I have two sessions on Shims set up for TechEd 2008, and I invite you to come and chat with me there. Also, I am in the running to be part of a Windows Vista Bloggers' Panel hosted by Mark Russinovich at TechEd IT Pro week. Come and barrage the panel with your most challenging questions - it should be a great time for all. There is still time to register - head on over to http://www.microsoft.com/events/teched2008/itpro/registration/regprocess.mspx to register.

Applying Shims (Compatibility Fixes) to Child Processes Using Layers (Compatibility Modes)

Chris Jackson — Mon, 19 May 2008 19:00:02 GMT

A comment came up on a recent posting regarding modules, inquiring about processes. Specifically:

Do these included/excluded modules have to be in the same process, or can they include other processes too? (I have an application that calls multiple child applications that all need to be shimed in the same way as the first process)

Ah, that is a subject I appear to have not touched on here.

Shims, you see, apply to the process you specify, and only that process. Layers, however, apply to that process and to child processes. So, if you have a main application that launches child applications, applying the fixes as a layer will help you achieve that (assuming they are always launched by the shimmed parent process).

Note that Compatibility Administrator uses slightly different terminology here. It refers to shims as Compatibility Fixes, and layers as Compatibility Modes.

Layers are also handy as a means of applying a collection of fixes. You can group several fixes together and call them a layer. But the fact that layers apply to child processes is exactly why we include a WinXPSP2VersionLie layer in the system shim database. A collection of one isn't that interesting, but a collection of one that gets the shim applied to child processes? That's very interesting, particularly for self-extracting setups that like to think they are installing on XP.

If the child applications can be launched directly by Explorer, then you'll want to shim them up separately and we can't save you the work by having you just leverage a layer.

System Shim Database Entries: They May Not Work, and How To Modify Them If They Don't

Chris Jackson — Wed, 14 May 2008 00:04:27 GMT

I received one comment on my recent blog post on copying from the system shim database, which I think is worthy of discussion as I have heard similar questions before:

I have found previously (under Windows XP, haven't attempted under Vista) that some of the built-in shims weren't suitable. I had a custom database for the applications into which I'd copy/paste/modify the system shim database. I don't recall ever seeing this error, so does compiling a modified shim not cause it?

To answer the question directly, I am not sure about Windows XP. However, on Windows Vista, even if I modify the shim, I end up unable to compile the SDB.

But that leads us to two discoveries.

First, the fact that we have shimmed something in the System Shim Database does not mean that it works 100%.

With some applications, we are simply shimming to get it to a particular point in the application. Working with one ISV, our objective was just to get it to hobble through to the auto update component of the application. They didn't care if it was a little bit ugly, so long as it actually got there.

With others, we have simply shimmed the application to fix the application and close a particular bug report that was submitted. If nobody ran the application to resolve that scenario, found that it failed, and submitted the bug to us, we haven't even tried to shim it.

Or, perhaps, we couldn't shim it, because the thing that it does wrong is unfixable using shims.

I have heard some people suggesting that finding the application in the list means that we are vouching for the fact that it works. We work really hard to get it that way, but unfortunately we can't make such a promise.

Second, you can modify the fixes that we have made to applications.

But it's not always easy.

If you have additional things to fix, then you can just add the application (don't copy/paste, because that's broken right now) and add the additional fixes that resolve the scenarios we clearly haven't tested before. We'll pick up both the entries in sysmain and in your custom SDB and apply both sets of shims.

If you need to remove an application fix from sysmain, you can right click on it, and disable that entry. If you need to disable an entry, we'd really love for you to tell us about it, so we can fix things up the right way. An example would be if you needed to add additional command line arguments to a shim. We can't just add the same shim twice or they don't both get picked up. Instead, you'd want to disable ours and add yours.

Why You Don't Need to Copy and Paste from the System Shim Database (And What Happens If You Do Anyway)

Chris Jackson — Thu, 08 May 2008 10:15:42 GMT

I hear about a bug now and again with Compatibility Administrator [CompatAdmin]. It always surprises me when I hear about the bug, because you can only hit it if you are trying to do something that there is no point in doing (which is why we never caught it before).

Here's how you reproduce the bug.

In CompatAdmin, expand the System Database, then expand Applications. Pick an application at random, right click, and select copy. Now, go to a custom shim database, and paste this application and its shims. When you go to compile the shim database (which happens when you save it), you get this really weird error:

---------------------------
Compatibility Administrator
---------------------------
There was a problem in executing the compiler. The database could not be created successfully
The database file might be write-protected.
---------------------------
OK
---------------------------

Now, the part about the compiler problem is correct - the suggestion that it may be write protection is not. So, we could do better with the error message, and we could also not give it in the first place. So, we have a bug filed against it. But here's the point.

You never need to manually apply shims from the system shim database. They are already there. They are already applied. We have 5,646 application entries sitting there waiting for you to install the application. As soon as we find the matching information, we start shimming - with no action required by you. We haven't just given you a collection of known fixes to discover and re-use, we've just outright fixed them.

Because, hey, when you go to run Barbie Pet Rescue, we don't want you to have to go fumbling around with CompatAdmin. We want to get you rescuing pets as quickly as possible.

What Modules Do I Need to Include when Configuring Shim Parameters?

Chris Jackson — Wed, 07 May 2008 07:51:53 GMT

A question came up in the comments of one of my posts, and I think the answer is important enough to elevate to more than just another comment (and I didn't even answer completely in that quick response).

When you are configuring a shim and press the "Parameters" button, we've talked about the "Command line" option for several shims, but we've kind of glossed over the "Module name" entry. When we're applying shims, we are modifying the IAT of all statically linked modules, and we also have to intercept calls to GetProcAddress for dynamically linked modules.

Which modules do we need to be sure to include? That's easy - just go one stack frame above the API call you want to intercept.

Which modules aren't included already? Well, that's somewhat more complicated. So, let's just walk through the list to see what might get cut along the way.

First, we automatically exclude anything that sits in system32. So, if you want to have something included that sits in this directory, you have to specifically include it. (That, by the way, is why you have to include msvbvm60.dll for shimming VB6 apps - because the runtime sits in system32, and most calls land in the runtime before they hit the APIs we want to intercept.)

Next, we have a global include and exclude list. Here, we can add back in some modules, or remove some modules from consideration that aren't sitting in system32. This list of modules is not documented anywhere. Conveniently, however, it turns out that there's a little bug right now that prevents the global exclude list from being processed, so the number of additional modules excluded is 0. But we may fix this bug (or we may not, if this causes apps to break because shims are no longer applied - I'm not sure about that right now), so it's good to know that this processing exists. I'll be sure to document the list if that comes to pass.

Then, the shim itself has an include and exclude list. You'll see that if you launch Compatibility Administrator with the (super-secret, undocumented) /x command line switch - here, we include modules to overrule global exclude list and system32 exclusion, or exclude modules that aren't in system32 or the global exclude list (were it being processed) which we want to make sure we don't shim.

Finally, the application of a shim to a specific program has an include and exclude list. You can include a module to overrule any earlier excludes (system32, global, or shim-specific) and exclude a module if there's something we didn't rule out yet where shimming turns out to be bad.

It would be interesting to have the equivalent of Resultant Set of Policy for shim modules, but alas, nothing of the sort exists today. Instead, you generally rely on investigating if a particular module calls the API and either does or doesn't walk through the shim code. But hopefully these rules can shed some light on why a shim may not appear to be wired, and adding in a module can provide a quick fix to resolve the issue.

The Lack of Wildcard Support for the CorrectFilePaths Shim in Windows Vista

Chris Jackson — Sat, 05 Apr 2008 16:44:25 GMT

A while back, I write about the Correct File Paths shim. Since that time, I've heard from a number of folks who are using the shim, helping them to configure the command line to fix applications (with the most common explanation being to "quote the pairs" - enclose each redirection pair in quotes, not each individual path, and then space-delimit the pairs you want to redirect).

But there is one specific scenario that seems to be cropping up from time to time - the lack of wildcard support. And where this really makes things complicated is in the root of C.

Say, for example, that you have an application that drops c:\myapplog_20080405_152438.log as the log created today, at this time. This happens quite a bit. Some people go down the path of using the command line "c:\;%userappdata%" and notice that this leads to the application being even worse off than before. That's because loading up anything off the disk - data files, program files, etc. is a file operation, and gets this shim. (You can configure somewhat the APIs that are intercepted for this particular shim.) Then, keep in mind that c:\ is going to match absolutely everything on the drive. Yes, it's going to match c:\myapplog_20080405_152438.log. It's also going to match c:\windows\system32\advapi32.dll. It's going to match c:\program files\my app\source data.mdb. You get the point. The application you shim with this command line is pretty much guaranteed to just crash and burn, because suddenly it can't find any files at all.

In this particular case, you have a workaround. You could use the command line "c:\myapplog_;%userappdata%\myapplog_ and we'll fix that right up. But what if it's truly completely random? What if sometimes it's c:\a1b2c3.log and another time it's c:\u3c8e5.log. What do you do then? Unfortunately, the lack of wildcard support means you can't just use "c:\*.log;%userappdata%\*.log" - and, in fact, you really can't use this shim. Which means you really can't fix this application using shims.

I've seen this enough times to open a bug, and I can get that looked into for a future full version of Windows. But with Windows Sustained Engineering, we don't like to change Windows between releases without justification of business impact. So, if the lack of wildcard support in this shim means you can't fix an application and you're going to end up delaying your deployment of Windows Vista, feel free to use the link on this blog to contact me, and send me (with your work email address) the name of your organization, the application (if it's commercial - otherwise you can just state the nature, you don't need to share anything proprietary with me), and the number of users who can't go to Windows Vista because you can't fix it. Or just post a comment here that expresses the value you'd see in having wildcard support. Whatever you feel comfortable doing - I just want to give you an opportunity to have a voice.