Follow-up from the research on how to disable a per-application shim. When I saw the output from the shim infrastructure in DebugView, I mistakenly assumed that the application was shimmed. It apparently only means that the application is found in the AppCompat system database. It does not mean that the shim is actually still in place.
Which leaves me with the question, how do I easily verify that an application is not shimmed (short of the earlier mentioned shotgun approach of turning the whole shim engine off)? Someone who does this for a living told me that one way you can tell is whether aclayers.dll is loaded in the process. He also told me that getting an application shim free is not that easy. There are all kinds of watchdogs in the system monitoring and instrumenting applications.
When an application entry is disabled in ACT, I see with Process Explorer two environment variables that indicate to me they have something to do with Application Compatibility or shimming: __COMPAT_LAYER = VistaSetup and SHIM_DEBUG_LEVEL = 9. The last one was added by me for the logging. I also see AcGeneral.dll and apphelp.dll loaded in the process. And it is wrapped in a job (you can tell by looking at the process properties in Process Explorer; there is a Job tab).
When I reenable the entry, I see __COMPAT_LAYER=VistaSetup WinXPSp2_GW and the same SHIM_DEBUG_LEVEL=9. This WinXPSp2_GW matches the entry in the ACT database. This somewhat confirms that disabling an entry does indeed have the desired effect. Also an additional dll was loaded: the before mentioned aclayers.dll.
When the PCA notices your application as a setup, it will flag it in the registry. This was my case and I was thinking maybe, the VistaSetup CompatLayer comes from there. I made sure the application was not flagged anywhere under (\AppCompatFlags\Compatibility Assistant\Persisted.) No difference. VistaSetup was still there.
Then I disabled the Program Compatibility Assistant service. The effect of that was that the process is no longer wrapped inside a job. However the service restarts quickly. I had to disable it to make sure it didn't get in the way. Still the VistaSetup was there as an environment variable.
My current assumption is that the VistaSetup layer is coming from the fact that there is no Windows 7 switchback entry in the manifest. It apparently is also detected as a setup. I will need to verify it by adding the switchback GUID. That is for next time.