On Tuesday 28 July we released guidance and updates to assist developers using our Active Template Library (ATL) to prevent creating controls or components with potential security vulnerabilities. Vulnerabilities in libraries are a rare, but industry wide issue, that requires broad collaboration and action by the community at large to effectively resolve.
Developers who have built controls using the ATL should take immediate action to review their control to identify if it is vulnerable and take appropriate action to rebuild their control using the updated ATL library and distribute a non-vulnerable version of their control to their customers.
If you develop any controls with ATL then please take a look at the guidance on MSDN detailing the steps required to identify and address affected controls. Also, we cover the issue in a Channel9 video.
This update causes major problems for those who have distributed applications based on the merge modules and wish to later patch their application. Once an application is distributed, you must obey the rules required for a minor update (no product code change), and this does not allow for installation redesign. Normally, you don't care to devote time to installation restructuring and testing in minor updates as well (which means you typically don't want to move to vcredist). Therefore, you need to be able to specify the version of the CRT in your build so that the installation package can continue to use the original merge modules and allow the automatic updating to get the newer versions installed. Therefore, it may not be that you want to use the old CRT, but rather that you have to use it for expediency.
Alternatively, you could conceptually merge all merge modules (e.g., RTM, SP1, and updated versions) into the installer for your minor upgrade as this would not violate any minor upgrade rules since it only adds new components from the new merge module. This would keep the previous compiler versions on the system after an upgrade is made but would remove them when the last referencing application is uninstalled. However, due to a problem in the VC++ merge modules that is trivial to fix (see https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=428663), this cannot be done either. This actually would be a good solution to this problem if this fix was made.
Thanks for the great feedback and comments. Just one further personal observation, I am not sure how much you gain (if you gain anything at all actually) by reverting the runtime libraries on your dev/test machines if you then deploy your binaries to customers who have machines accessing the internet or on which customers are installing other products. Installing any later/higher minor version of a major version of the VC runtime libraries, either by MS doing an update or customers installing another product (or by customers just installing a later vc_redist* themselves too), will cause your application to run on the later/higher minor runtimes (regardless of what you build/tested against.) So for testing (and therefore your deployment) in most/all cases I would think you could argue that you want the latest version available (since that is what your binaries will most probably run against on end-user machines.) Of course if your end-users cannot update/install software then this is not an issue. As an aside, this is also why we aim for compatibility between updates too, otherwise we too would have to test/redeploy many of our MS line of products whenever we updated the runtime libraries (well of course we do test them but rarely, if ever, do we need to redeploy anything just because the VC runtimes were updated – but we do want to be sure.)
I am adding my comment only because I as yet do not see any responses from this blog's authors that represent an understanding of the issue.
There is a fundamental difference between applying security fixes to software and changing development environments.
It is unacceptable that a security patch updates other applications development environments.
I found reading this blog entry and the nonresponse to the central issue very frustrating.
I would like to read that Microsoft understands their mistake and that they will never do this again.
Bottom line Microsoft now has a trust issue. Until I hear Microsoft acknowledge this and pledge to change, the only alternative is to disable Windows Update to protect against these damaging updates.
Make no mistake changes that invalidate testing, cause support issues, wasted time researching, reading and responding to blogs, and the delayed releases, etc. that have mentioned in this blog it is obviously a damaging update.
Does Microsoft get that?
I will echo the previous posters in that the blog's authors do not seem to understand the issue. We are ONE DAY away from releasing our next version of software, and quietly, unexpectedly, and sneakily, Microsoft go and mess with our development environment behind our backs.
Suddenly a build that worked perfectly a few days ago is filled with errors and worse, we can't explain why. Please do not ever do this again. You simply cause trust issues - we will now be deactivating automatic updates across all machines simply to avoid this issue ever happening again. Is that really the response you wanted? Is that really how you want your critical updates viewed?
Have you perhaps forgotten that people need to update installers as well to fix this mess, and that doesn't happen by magic?
Just to follow up on this specific reply "Re: your point 2) building on different configurations to those you develop on is again your choice (and given a large organization with many developers occasionally seems inevitable to some degree) – but I am not sure I see the issue here, if the build machines are consistent..."
They aren't in all cases. Large software projects inevitably pull in plenty of third party (sometimes internal) libraries that were built for VS2005 RTM or SP1 and cannot be rebuilt at a whim (major costs involved). Thanks to this update, Visual Studio then proceeds to embed a default manifest referring to both CRTs/MFCs, which certainly seems incorrect (please say if it is correct, because there's advice out there to the contrary), and the entire thing occurred with no warning until our automated tests broke because we had no idea that we needed new redistributables!
I'd be the first person to say I appreciate bugfixes and updates in older products, but the lack of proper communication over this is completely unacceptable. Phaeron hit it on the head with his list of failures so I won't repeat them.
I think Visual Studio needs to be prepared before those updates can take place without hassle:
1) Make the runtime selectable from the project settings, graphically, e.g. a combobox, not only by defines which nobody finds if one does not know they exist. A runtime security patch should never ever change that project setting. So only the project owner decides if any newer runtime should be used.
2) For being able to target different runtimes and create setups for and with those runtimes the VS setup and patches should allow "side by side runtimes" (libs, dlls, pdbs, merge modules and vcredist) so that on each machine you always have all versions of the runtime available, not only the latest. So I would expect each set of runtime files to be put in its own redist subdirectory, so no newer runtime replaces an older one - I do not talk about windows\WinSxS but about the VS files under programfiles.
3) With the above "versioned" runtime directories the built in setup projects could find their correct runtime merge modules as well, so neither new builds nor new setups cause any change in runtime distribution.
With the above prerequisites I would not mind too much to get new runtimes installed by windows update.
"Large software projects inevitably pull in plenty of third party (sometimes internal) libraries that were built for VS2005 RTM or SP1 "
to add to JeffH's reply, I can give a real world example: ActivePDF, which provides ISVs with PDF capability for their apps, ships a redistributable DLL that is hard linked (in the DLL's manifest) into the 762 versions of VC2005SP1. With policy redirection, etc, it should be ok (can't be 100% sure though), but I obviously don't like having mixed references across binaries in a shipping app.
Yet another dissatisfied customer.
1) My app is linked against the installed VS CRT. I ship the installed VS CRT. My exe requires, and is distributed with, 3rd party DLLs which are built by the supplier. The 3rd party DLLs reference a specific CRT verision which is supposed (!!!) to match my VS2005SP1 CRT. If the CRT version silently changes then I have two problems: (a) I'm shipping DLLs whose manifests require a different CRT version from my exe, (b) I'm only shipping one CRT version but I now have a dependency on 2 versions.
2) I do not expect an automatic security update to silently break my build deliverables. ever.
And I have just been bitten by this a second time in a month. Once again my beta testers are complaining of broken builds. Somehow the update is back even after I disabled autoupdates, perhaps some OneCare issue?
Hello JeffH and others
Thanks for your input and your feedback. Many of us are reading the replies here and we do take your comments very seriously when reviewing our procedures and processes. In order to provide our feedback I wanted to state that, at this stage, I see nothing here that will cause us to make any changes to our processes for handling VC runtime library security updates going forward. This means that if we find high priority potential security vulnerability in one of our libraries we will update the libraries and release update(s). We feel we have a paramount duty here to protect users and those who ask for updates (either manually or automatically) will receive them – this is generally our highest priority. This action will normally not affect binaries currently running on end-user machines – these binaries will run on updated VC libraries but we strive for no “breaking changes”, as we achieved with the ATL Update, and everything should just work for existing binaries. Yes, we may make a mistake with this from time to time but if so please let us know and we will fix it if at all possible. Of course sometimes the Update may force a breaking change – but again if that is the only way to address a security issue then that may be what we have to do. If anyone did get broken in this scenario with the ATL Update (their already deployed applications broke) then we are sorry and we would like to know about it.
The other salient point is for binaries built after the update with the new VC runtime libraries had been applied to a development machine, here I want to restate that we only support a model where the libraries on the development/test/end-user machines are synchronized. Here synchronized can mean that downstream you can have later versions but you must ensure that at a minimum they are all the same version. You can achieve this a number of ways (different mechanisms come with different requirements and risks) but you must ensure that this condition is satisfied. Our general recommendation that is you deploy the VC runtime libraries that you build your binaries with alongside those binaries– in this case you should always be fully protected. With regards to the ATL Update case, the update to developers’ machine should only have trigged warnings/errors if you have code which potentially exposed the vulnerability. If you got build breaks then you were at risk. Sorry if you got caught in the last few days of development but I think it could have been worse. Testing/deployment breaks can only occur if you build on an updated version of the runtime libraries but did not synchronize them on to your test/end-user machines. Again if you followed our recommendations we believe you would have been fine in this scenario too however if you have experienced an issue then please let us know.
With security updates we do follow a set of procedures related to disclosure. We do have to balance exposing the issue (to both the good and bad guys), along with the current level of exploitation of the issue, along with the risk of the exploit being publicly exposed by a third party. Without going into the details we followed our normal process here of issuing an Advanced Notification three working days before deployment in this case. On the day of issuing the Security Update we released numerous online details about the update (blogs, videos, etc. – the links are throughout the thread here.)
As always, please post any follow up comments you may have here and we will read them.
> In order to provide our feedback I wanted to state that, at this stage, I see nothing here that will cause us to make any changes to our processes for handling VC runtime library security updates going forward.
I strongly urge you to reconsider your position. By distributing a critical security patch and installing a redirect to that version you would have already accomplished the goal of protecting users. All you have done by forcing a dependency upgrade during the compile process is break builds across the globe. Doing this is directly counter to your stated goal of protecting users because (a) you have convinced people that accepting automatic updates is an unacceptable risk and (b) you have made the statically linked version of the runtime libraries a more attractive option, which you cannot service.
Thanks for the feedback – again sorry that this causing you such pain, it is definitely not our intention.
For issue 1, if you ship the latest VC Runtime Libraries with your application you should be fine; all the libraries your application depends on will be rolled forward to the latest minor version for all dependencies. (See my somewhat related previous comment: “Installing any later/higher minor version of a major version of the VC runtime libraries, either by MS doing an update or customers installing another product (or by customers just installing a later vc_redist* themselves too), will cause your application to run on the later/higher minor runtimes (regardless of what you build/tested against.)”). Again if you break with the latest version then please let us know, we aim for no breaking changes when we do updates but we can make mistakes.
For issue 2, the VC Runtime Libraries you build against (or a later version of the same) need to be installed on the test/end-user machines for deployment to work. This is the only model we support. If you deployed the relevant version with your application and it fails then we would like to know.
With all do respect, you are not understanding what Ross is saying.
That could quite well be the case - sorry about that. Can you and/or Ross restate the issue you have? (And I will look at Ross’ post with a couple of others from the time and see if they can tell me what I have missed.)
I really do appreciate you taking the time to raise your comments/concerns and as I said we are discussing this internally and value your feedback and input. By the way, I did not mean to imply that anything I say is absolutely final. I just wanted to be clear that at this stage we were not changing the process – I think we also need to be transparent and give you direct feedback - but do not take that as we will never change the process. Obviously many of you feel quite passionately about this.
Thanks for your attention. I'm going to put aside my opinions of your management of this issue and try to help you understand my requirements.
I ship an exe bundled with 3rd party DLLs. As I understand it, all exe+dll manifests need to reference the same crt version. At the moment, the way I achieve this is to use the correct version of MSVC (in my case 2005sp1) to build my exe. Until this update, using my versioning method of using the correct MSVC version gave the desired effect (my exe referenced msvcrt 8.0.50727.762). After the update, my exe msvcrt versioning was broken. A basic fact of my use-case (and many others) is that I do not have the ability to determine the msvcrt version referenced by the 3rd party dlls I use. Since (afaik) the SxS mechanism requires all dlls+exes to reference the same msvcrt version, I need a way to achieve this goal. -- more to the point, I need MSVC to allow me to specify what version of the crt I use. This update breaks the previous widespread assumption that installing a specific MSVC version (including specific service packs) would give me msvcrt manifest compatibility with other dlls built elsewhere based on the same MSVC version. With your update I need to use 3rd party DLLs referencing VC2005Sp1+kb971090 (which don't exist).
I think one possible solution for this issue would be for MSVC to support multiple side-by-side installed crt versions for development (multiple .libs etc) instead of just one version which gets blown away with each update. That way I would have the option of referencing a specific msvcrt version in my builds if needed (or just using the latest version by default). I think this would be an improvement over your current approach and one I would like you to seriously consider for the future.
I notice you reference this blog post above:
Is this method documented in any Microsoft documentation? It seems like it might partially solve my problem. From what you say, if I make my exe manifest reference msvcrt 8.0.50727.762 using this method, but ship 8.0.50727.4035 then I should be ok. Is that correct? Can I use that method even if msvc links my exe against 4035?