I wanted to post a response to what I consider to be somewhat suboptimal advice, with the hopes of correcting it.

In recorded history, there have been a number of classic blunders when it comes to advice. They include:

“Hey, let’s start a land war in Asia!”

“I really think you should buy the Betamax instead of the VHS.”

“Let’s go a little faster. Nothing can sink this ship – it’s the freaking Titanic!”

And now, to add to that list, is my friend Greg Lambert’s latest advice on shims: http://aokcompat.blogspot.com/2010/10/shims-those-nasty-bits.html

Before I get into a discussion on this, let me make sure that my position is absolutely clear: nobody looks forward to using shims. It’s not something that I recommend that anybody go out of their way to use, or keep around any longer than absolutely necessary. It’s not something that you have to use in order to be successful, nor is it necessarily your first choice.

But, if you want to manage costs of your deployment (who doesn’t?) then I strongly recommend keeping this option in your toolbox. As you troubleshoot compatibility problems, you’re going to build a list of technical options for what can be done when something doesn’t work. You can, of course, always buy or build a new one, but that’s the most expensive choice. (It’s also the best choice in the long run, if you’re going to be keeping the app around for a significant length of time – but it’s often nice to divorce the timelines of this option from the timeline of your deployment.) You can also virtualize and continue running it on a previous operating system, but then you have to manage that operating system and buy machines with enough resources to run it (also not free). After that, you have a list of things which may work. Maybe you can shim it, maybe not. Maybe you can modify the MSI, maybe not. None of them is a catch-all, so if you remove an option, you’re going to be able to fix less, which means you’re going to be forced to select one of the more expensive options. Unless your management gave you a budget of $∞.00 you may not want to exclude a viable option.

In my experience, people include this option technically, and then build business rules to include it from a business perspective in cases such as:

• The update is going to take too long, so I just want to buy some time
• The code with the bug was replicated across a number of our apps – rather than waiting to fix them all, I can deploy today and have the shims roll off as they gradually upgrade these apps to fix all occurrences of this bug
• The company is out of business, so we need time to build or buy an alternative – it’s not clear what that is today, and I have no other way to fix it in a cost-effective way

I do recommend thinking through and determining the scenarios where it’s OK for your project team to use shims, because they are definitely a duct-tape solution.

So, with that bit of an intro, let’s examine Greg’s points:

Microsoft owns the Shim database and may update the database at anytime

I admit, I don’t understand this argument. The fact that we use shims to resolve compatibility issues doesn’t seem like the most compelling issue for you not to use it. In fact, since this what we primarily choose to fix app compat issues, it seems like a fairly strong endorsement for it. Keep in mind, we don’t own all shim databases – we own ours. sysmain.sdb contains the commercial software we’re fixing, and of course we’d like to be able to fix more applications that we weren’t aware of when we shipped Windows 7, because people like their software to work whether or not we happened to have a copy or a bug report prior to July 22, 2009. If you make your own (one or more), it’s yours, and we don’t touch that. We’re not going to overwrite your work.

No published API’s for the Shim Database

This argument is not exactly true. There are published APIs for the shim engine. They are here: http://msdn.microsoft.com/en-us/library/bb432182(v=VS.85).aspx. However, it would be fair to say that there are no easy-to-use APIs for managing shim databases. This affects Greg from the perspective of building a tool to automate fixes, making his job unnecessarily difficult, and it’s something that we’re actively investigating to make his job as a partner with Microsoft easier. However, it’s not something that affects the enterprise – I am not aware of any blocked scenarios in the enterprise as a result of the lack of easy-to-use C APIs for the shim engine. So, I don’t consider this a compelling argument for enterprise adoption, though it’s still an action item I’d like to do a better job with.

No published documentation for the Shim Database schema

Same comment as the above for APIs – it doesn’t affect the enterprise. We also don’t publish all of the internal structures and code used in structured exception handling, and we still recommend that you use that. I believe this is also a result of conflating the needs of the ISV from the needs of the enterprise.

No tools for managing the Shim Database in an enterprise environment

While our tooling could certainly be improved, we provide tools for creating (Compatibility Administrator) and installing / updating / removing (sdbinst.exe) shim databases. We are very interested in hearing about the scenarios where better tooling is required. But, since this is a duct tape solution, it’s something we hope you aren’t actively managing day to day, but rather something you put in place initially and hopefully never have to touch again, or at least only have to touch once or twice more. If you’re routinely fiddling with shims, that’s admittedly a bit unexpected. We are very interested in hearing expectations and experiences as adoption of this mitigation grows.

No Active Directory integration

There are many things which are not AD integrated, yet are still a good idea. Pecan Pie is not integrated with AD, yet Pecan Pie is freaking awesome. Personally, I’d like to see a GPO-based solution, but it’s not completely clear how that would work, or how it would work better than what you would do today, which is through pushing a script. So, this seems like the same argument as the last one: management exists, but could certainly be better. I agree, but don’t consider that an insurmountable obstacle. IT Administrators do many, many things with scripts today where GPOs don’t cover specific scenarios.

No native integration with 3rd part deployment tools

Same as above – any tool which supports scripting can be used, and management requires a single line of script. Any explanation of the specific requirements of the enterprise can help us to focus investments. We don’t want to over-engineer what we hope to be a very temporary solution, but we want to do enough.

No central management and control of the Shim Database

Same as above.

No centralized reporting mechanism’s available (from MS or ISV’s)

Same as above. We do have instrumentation, as documented here: http://blogs.msdn.com/b/cjacks/archive/2008/05/20/enabling-diagnostic-output-from-shims.aspx

Any update to the Shim Database requires a complete re-compile of the database

I’m not sure why this is a problem, per-se. First, I’m curious as to how often people are updating them – for my customers, it’s not terribly often. Second, since a recompile is so fast and can be done with any computer, and starts off with the ability to reverse any existing database (installed or not) so source code maintenance is not required, I don’t understand why this is a blocker. In fact, I don’t understand the scenario (outside of the ISV scenario) where you would want otherwise?

No transparency or “accounting” of changes or updates to a Shim database

Shim databases are completely transparent – you can view them on any computer using the Compatibility Administrator tool. Now, when you update to a new version, there is no reporting on the delta, so you’d have to manually compare, but it can be done. But my opinion here is that it’s relatively transparent, of little interest to the end user, and the administrator who deploys the update should be completely aware of what changed, since they, after all, were the ones who changed it. Perhaps I don’t understand the scenario well enough?

No “back-out” or roll-back facilities for the Shim Database

You can uninstall a database with sdbinst /u very easily. You can also roll back to a previous version by calling sdbinst and pointing at the previous version (SDBInst will overwrite any custom shim database with the same GUID, whether moving backwards or forwards).

Limited Multi-Platform Support

I would go as far as to say zero multi-platform support. While you can create an SDB on a different OS than the one you deploy it to, it’s generally a bad idea to do so. While the shims you apply may exist, the semantics might change. And, if they don’t exist, it won’t wire up anything. But I think this indicates a misunderstanding of the intent of shims. They are not designed to be portable. The shim database is an extension to a particular OS to make the applications multi-platform by fixing the bugs that stop them from doing so, and shouldn’t be carried around between OSs. An app may run natively on Windows XP, need one set of shims to run on Windows Vista, and a separate set of shims to run on Windows 7. You don’t package the shims with the app and expect them to work everywhere, you package the shims with the OS so the apps that you want work on the OS that you want. We seem to be associating the shim with the wrong entity – they belong with the OS, not with the app.

To support these points, Greg then quotes me:

It is possible that you will eventually find that the shims assigned to resolve a set of compatibility issues in an application are not comprehensive and that later you will need to deploy an updated version of the custom shim database that resolves the additional issues your organization later discovered. If you deployed the original custom shim database as part of the installation package, you will need to locate each client that has installed this application and the original custom shim database for it to replace it with the new version.

It’s fairly important to take this quote in context, so let’s apply the context and clarify this quote. There are a couple of primary choices people look at for deploying shims: deploy them with the app, or deploy them with the OS. In most all situations, I personally believe that the better choice is to deploy them with the OS, and manage a single central shim database rather than a number of stand-alone shim databases containing a single app entry which you deploy with each installer. While that method is OK when you only deploy one or two and in general seek to avoid them, as soon as it goes even slightly behind a one-off, I believe it makes sense to have a single, centralized shim database for the organization based on the input from many customers I have worked with.

So, it’s somewhat disingenuous to take a quote in an argument against choosing a particular methodology of deployment and generalize it as an argument against using the technology. That would be like declaring that deploying Windows 7 is a bad idea because deploying it via sneakernet (having a guy walk around with a DVD and manually install it) has its disadvantages for a multinational corporation.

Greg then points out his recommendation, which is to use “Best Practices offered by Microsoft”:

Elevation Manifests

It is emphatically not a best practice to apply an elevation manifest to somebody else’s code. Under no circumstances do I believe this is a good idea. In suppressing or instigating an elevation prompt, the very act of adding a UAC manifest disables many compatibility features in the operating system. For example, you lose file and registry virtualization. That means you have to manually remediate many of the most common file and registry permissions issues. Since the argument is to not use shims, then you are left with the old Windows XP style of remediating these applications: by modifying permissions. There is a reason why not many customers migrated to standard users on Windows XP: because migrating without these assistive technologies was too difficult and expensive.

Please, do not apply a manifest to code you have not written. This is an anti-best-practice in the application compatibility project scenario, and will potentially lower security and definitely generate more work. Apply elevation flags via a shim database instead.

Updating the source application package (MSI Installers)

Absolutely! Depending on the customer, statistically the average number of application issues that are installer issues can be up to 40%, sometimes even more, so you have to fix things here in the package. There are also a number of issues where you can drop working code and remove the buggy code, which is definitely preferable to shimming buggy code when that is an option. I enthusiastically agree, but believe that this is orthogonal. There are many things you can’t fix with shims which you can fix by changing the package, and there are many things you can’t fix in the package which you can fix with shims. It’s good to have both tools on the approved list, and if both are an option, I’ll almost always choose this one! Greg is spot on here.

Creating Side-by-Side (Sxs) Isolation manifests

In my experience, this resolves a relatively small number of potential issues, so I seldom find this one drifting to the top of my list. However, it is another potential tool which I keep in the back of my mind when it comes up as appropriate, and my anecdotal experiences around frequency aren’t necessarily representative.

In summary, while I don’t necessarily advocate the opposite of Greg’s advice and wouldn’t say that you absolute must use shims. Instead, I recommend that you don’t rule them out straight away. Rather, I’d consider them a valid technical option, and then make the business decision for when you choose to utilize this, among other, technical options. Doing so can help you to minimize cost and accelerate your deployments. They’re simply too valuable in too many situations for you to avoid them outright because of fear, uncertainty, or doubt.

A customer came to me with a question: they no longer saw the “Size” column in the Programs and Features list (where you go to modify or uninstall software) on Windows 7 when the installer was not an MSI.

Here’s the story: we used to run some heuristics, as described here: http://blogs.msdn.com/b/oldnewthing/archive/2004/07/09/178342.aspx

However, if you read through those heuristics, you can see that they’re not exactly the most precise, and in general it’s often better to not say anything than to say something wrong. So, the article then went on to suggest that you populate the EstimatedSize value in the Uninstall registry key to avoid the guessing.

Which, of course, had one problem: we didn’t always look there if the application wasn’t installed by an MSI. (A web search will find some of the clever workaround that people have discovered to get ARP to look there.)

For Windows 7, we fixed that problem: we now look at the EstimatedSize value for any application on the system. Furthermore, we removed the heuristics altogether (they were pretty hacky) and only look at the EstimatedSize value.

Conveniently, the customer who brought this up happens to make programs which are used to create application installers. They’re big enough that I’m pretty sure you know their name. So, my guess is that they’re going to start writing that value in any non-MSI installers which aren’t doing so today.

My friend Aaron over in MCS Federal posted a recruiting blog post a little over a week ago here: http://blogs.technet.com/b/fdcc/archive/2010/10/02/job-opening-app-compat-guru.aspx.

Well, he’s gotten a pretty good response back – with more candidates that look pretty good than the federal team currently has headcount for. So, I started looking around in the Commercial business (which, unlike federal, doesn’t have the same requirements around U.S. Citizenship) to see if we have headcount (I knew we had demand) and it turns out that I can find some also.

So, we decided that we would pool our resources to do a single round of app compat recruiting. If we can possibly coordinate it, we’re looking to get together a big pile of folks doing app compat within Microsoft to run a single, consolidated app compat finals day.

Interested?

Here’s what makes a great app compat gal or guy at Microsoft:

• Ability to deal with bulk, and help whittle down a large pile of potential problems to a small pile of actual problems and a plan to address them
• Ability to talk to developers, designers, IT administrators, and business users
• Ability to remove reliance on a single tool and instead apply a good process to a collection of tools, bonded together with human intellect
• Ability to persuade people without authority
• Knowledge of Win32-level details of how Windows works, and/or wire-level details of how IE works – in other words, a general ability to get to the bottom of things

Ah, yes – that last one, it’s a kicker. See, when people call Microsoft for help, they don’t just want someone to set up yet another ACT server, they want someone who can get to the bottom of things. Who, when necessary, is going to be able to whip out a debugger and figure out what’s going on. Someone who intuitively is able to look at a set of baffling symptoms and start narrowing things down quickly.

Those who have been most successful have traditionally spent at least a bit of their career writing Win32-level code in C/C++. Not necessarily in their last job, and not even necessarily for commercial software. But, if you are the sort of person who even tinkers with that for a bit just because you just have to know what’s going on under the covers? Perfect.

But, what’s in it for you? That’s the hardest part of our job. Taking someone who can understand how code and platforms work at that level, and then convincing them to take a job not writing code? Sometimes not so easy. But, for everyone who I’ve talked into this, it’s been a win. The feedback I hear from alums of app compat tours of duty include comments like:

• “Helped me really understand how Windows and IE work at a very deep level – I’m a much better developer today as a result.”
• “I learned to think of software as living in a diverse ecosystem, and saw the complexity of managing the next 100,000 installs of that software instead of just the instance on my machine.”
• “Working with app compat made me appreciate the importance of designing software to last for a dozen or more years, since that’s what seems to happen an awful lot.”

Sound awesome? It’s had me hooked for 5 years and counting. If you’re interested in taking on a challenging position which is in incredibly high demand, further mastering your understanding of one of the most complex app platforms in the world, and coming out the other side a better developer and/or technologist, then let’s get your name in the hat. We’re hoping to get our finals day together soon. Both Aaron and I are specifically aware of US-based positions, but we’ll forward along your resumes if you are elsewhere, as the demand is definitely worldwide.

appcompatguy at microsoft dot com, or use the email me link on this blog. Happy to answer questions or get your name into the hat. Hope to hear from some of you soon!