I receive a number of questions on the compatibility evaluators in ACT that revolve around one central question: what are they actually good for?
Seems kind of a harsh question, eh? Well, I’m not intending to be rude. However, I do try to help people avoid assumptions that will end up making them sad. However, I have since discovered that, in my attempt to prevent you from spending a lot of money and ending up sad, I’ve erred in the direction of leaving you sad but with your money still sitting in your pocket. I guess that’s slightly better, but I’d rather you weren’t sad.
You see, a lot of people approach the Application Compatibility Toolkit with a perspective of reverence. I mean, look at the name! It’s made by the Windows team! It has to be all I need to get the job done! (In fact, if you are an ACF partner, I believe it’s even mandatory to use it.) But, if I choose to use the compatibility evaluators, what can I then do with that data?
Well, people initially assumed that they could run the evaluators and it would tell them which applications are broken, and which are not. They could then use that data to project costs for the project. Like so:
Run Evaluators | V Project Costs | V Fix all issues the agents flagged | V Ready to deploy!
And, if you do that, you end up sad. Because we’re going to let you down. We don’t find all of the apps that have problems, and we don’t find all of the issues in the apps we do flag. We are runtime evaluators, so we have to be concerned with performance. Even if we could look for all bugs (hint: we can’t), in production your users would hate us for making their apps miserably slow by looking in too many places if we tried that. So, unless your app just so happens to have a bug that’s extremely common, we won’t even notice the bug.
So, why do we have these evaluators that don’t help you either project costs or find all of your issues? Is it because we don’t know how to write programs? Nope. (Not this time at least.) You see, there is a really good use for this data, and if you pick this use, then you not only end up unsad, you may even end up happy.
Issues detected by compatibility evaluators come with a priority automatically set. We only set it to Priority 2 or Priority 3. (Setting it as Priority 1 – critical to fix – is something left for you.) What this means is that, with a Priority 2 bug, you have an application bug that is probably not automatically fixed by the OS, in a bit of code that somebody actually ran, so you probably want to fix that. If we flag it as a priority 3 bug, then it’s still a bug in a bit of code that somebody was actually running as part of their job, but it’s something that’s probably automatically fixed. For example, UACCE will flag file writes. If we predict that UAC virtualization will fix it automatically, then priority 3 (nice to fix). If we predict that UAC virtualization will not automatically fix it, then you should consider fixing it so we classify it as priority 2 (must fix).
So, I’m not so much interested in seeing the original estimate (since we miss so much stuff), but the data does come in handy down the line. For example, here is a segment of an application testing workflow that incorporates this data:
Perform Install Testing | V Any Sev2 Issues? –yes—> Remediation | no | V Perform smoke testing, user testing. etc.
Now I’m using this data in a productive way to save time from a manual effort. You know that some user ran into this problem while performing their actual work, so the data fidelity is very high. Why send a known broken application over to testing and waste manual testing hours discovering a bug you can discover with nothing more than a few mouse clicks (or perhaps may outright miss if you don’t have a good test script)?
ACT agent data is relatively inexpensive to collect if you need the inventory anyway. But, you need to avoid getting tricked by overly optimistic sales people into believing that this data is everything you could ever want, but at the same time make sure you don’t ignore valuable data. Runtime data is awesome because you know for a fact that bad thing actually happened, and if run in production you know it happened as part of doing your real work (which are the only bugs you care about).
Feed your workflow, save manual effort, and reduce your risk. Now that is what ACT agent data is good for.
And, of course, we certainly do wish that we could highlight all busted apps for your organization (to help you better estimate project cost), as well as discover all individual app issue. Static analysis tends to do a better job at the app level (is the app broken – yes or no?) simply because it can perform WAY more tests without interrupting somebody’s work. But it doesn’t do nearly as well at the issue level. In the end, a balance between runtime tools, static tools, and manual effort is what most people end up doing to build the plan that really works for them. Bringing it all together, you can build the optimal mix of low cost and reduced risk during an app compat project. Don’t ignore a component of your solution just because it isn’t perfect. Because, alas, it’s all not perfect. There are no silver bullets. But we do have a few lead ones.
This is a question that has been coming up a lot lately, so I figured I would answer the question once here, so I can just start handing out the link when it gets asked again in the future. Here is the essence of the question:
“I am trying to create a process as another user, using CreateProcessWithLogonW. Even if I launch it from an elevated process, it spawns a Medium IL process. If I mark it as requireAdministrator, it returns ERROR_ELEVATION_REQUIRED. What gives?”
First, let’s explain why you always get a filtered token. When you are launching an application using this API, somewhere behind the scenes you are logging on that user. With UAC enabled, if you are a member of the local Administrators group, that’s going to result in generating two different user tokens: your full administrator’s token, and a filtered administrators token. So, you’ll end up with both logon sessions (and tokens) generated, but then we’ll use the filtered one until you explicitly elevate.
OK, so far so good. The logon event itself causes me to generate both logons and use the less powerful one by default. (Assuming it’s an interactive login, that is.) I get that, so now I’m going to manifest my application to get around that and use the more powerful one, and I bounce into failure again (and failure sucks). Why is that happening? ERROR_ELEVATION_REQUIRED? I mean, come on – I know that, that’s why I’m trying to elevate!
Well, there’s a good explanation for that one too. This is based on the layering work we’re doing in Windows. Rather than letting any Windows binary depend on any other Windows binary, we invest huge efforts in maintaining the architectural purity of layering in Windows. Binaries that are at a low layer just can’t depend on binaries in a higher layer.
Larry describes the layering process here: http://blogs.msdn.com/larryosterman/archive/2005/08/23/455193.aspx.
What’s the layering problem? Well, CreateProcess is really low in the layers. What can you do without the ability to create a process? Not a whole lot. Elevation, however, is a different story. It requires a trip to the app elevation service. This then calls into consent.exe, which has to know how to read group policy and, if necessary, switch to the secure desktop and pop open a window and ask the user for permission / credentials, etc. We don’t even need to take all of these features, let’s just take the dialog box. What can you do on a system with processes but not a CreateWindow function? Why, be server core, of course!
Now, for creating a process that requires elevation, normally you just switch up APIs. The shell sits in a much higher layer, and consequently is able to take a dependency on elevation. So, you’d just swap out your call to CreateProcess with a call to ShellExecute. So far, so good. But wait … if you do a web search on ShellExecuteWithLogon, you return 0 hits. That’s right, there is no API for this!
So, how can you work around this? You need a bootstrapper. Some process which will let you do the transition to the alternate user, which could be responsible for running the requireAdministrator application. So, you could design something like this:
We did something very similar to this for Sysinternals’ ShellRunAs, only instead of writing a bootstrapper, Jon used cmd.exe to bounce through to the target executable (which is a technique you could use as well, though Mark had to do a bit of work to try to keep this hidden wherever possible).
(Why did we use cmd.exe? Because there’s a lot of logic built into this already, such as handling different file types and targets, which would have been challenging to get as correct as this already extensively tested application.)
Why don’t we just create the ShellExecuteWithLogonW API? I’ll never say never, and we might at some point. But today, the use cases for this APIs have been use cases where there has been an alternate design which is superior. The most common request is for people writing home-grown software deployment software, where they’d like to encode credentials right into the app and elevate their own process. The real problem here is not the lack of the API, it’s that you have admin credentials encoded in your app for the world to read. If you have that, what you want is a way to get to a place where you do not have that as quickly as possible, not make it easier to build on that design. I’ve also seen scenarios where moving the code in question to a service was the better design. (Adding new APIs is a big deal, so we tend to look for whitespace with no workarounds if we’re going to do that.)