Grr... My VC++ Project Is Building Slower in VS2010. What Do I Do Now? (A Step by Step Guide)

Grr... My VC++ Project Is Building Slower in VS2010. What Do I Do Now? (A Step by Step Guide)

  • Comments 13

Renin John

Hi, my name is Renin John and I work as a Software Development Engineer in Test with the VC++ Project and Build Team. I was involved in testing the Build performance of the VS 2010 IDE and I wanted to give you all an idea about what to expect, and some basic steps you could take if you end up in a situation where your project builds slower in VS 2010 than it did in VS 2008 or a previous version.

VS 2010 has better or equal performance in majority of the cases we have measured. However, if you’re hit by a build performance degradation, stay tuned and I will show you ways to investigate the issue, identify whether you are comparing apples to apples, figure out a way to improve the build performance and report bugs with appropriate level of detail so we can take action as quickly as possible.

 

I - Identifying Your Situation

  1. What kind of build am I doing?
    • If you just start up/restart your machine, open VS and do a build, it is said to be an Arctic Cold build.
    • If you close down VS, open it again and then perform a build, it is said to be a Cold build (since memory is cached in this case).
    • If you did a build, and you’re redoing it (Clean + Build, or Rebuild Solution), the one you’re redoing as well as any build you redo after that, are called Warm builds.

      Performing a Rebuild is similar to doing Clean Project 1, Build Project 1, Clean Project 2, Build Project 2, etc. Hence be aware that there is a cleaning overhead involved in the build time.
      Also make a note of the configuration you are building in: Debug, Release, … The performance will vary due to the different switches that get passed to the tools in each case.
       
  2. What is my Build Output Verbosity setting?
    The output verbosity level impacts the build time depending on the amount of text it needs to print out to the console and the log file(s). When I did a comparison, building Diagnostic took about 5% longer. If you’re not already aware, this setting had no impact in VS 2008 … the outputs generated were always the same.

    What is my Build Output Verbosity setting?
     
  3. Did I make a Remote Desktop Connection to the build machine?
    If you connect remotely to the machine to do a build, you will observe a noticeable difference in the build time when you compare it with what it takes to build locally on your physical machine, based on the network bandwidth, latency, etc. A cause for this variation is the build text being constantly outputted and scrolling taking place, in the Output window.
    Therefore, take care not to compare the build times of (say) a VS 2008 solution build locally versus a VS 2010 solution build remotely.
     
  4. Am I running a virus scanner?
    The anti-virus software that you’re running on your machine could interact with the tools during the build process negatively impacting the performance of a build. The level of interaction might vary between different versions of VS.
    During the development process of Visual Studio 2010, we tested the performance of our build scenarios with different anti-virus software installed. We were able to measure notable performance overhead in some of these scenarios and worked with vendors to help us understand and improve our toolset. One interesting thing about AV software is heuristics and scanning algorithms change frequently as new viruses and attacks are discovered in the wild. With that said, the performance we were able to measure at the time we shipped Visual Studio 2010 was acceptable with respect to the presence of AV software, but those characteristics are subject to change as AV software continues to evolve.
    While we are working continuously on making our tools robust in the face of security scanners, you could consider disabling your anti-virus from scanning the build output directories or entirely, for better performance. Microsoft cannot recommend you disable your anti-virus software to improve performance, but the potential interaction with it is something worthy of consideration.
     
  5. What other services do I have running on this machine?
    Like the security scanner, other services could also be having an impact on your build time. If you see the need, take a few minutes to identify and turn off any services that you are not using before doing the build. In addition to the anti-virus mentioned earlier, some other services you might want to turn off include Superfetch and Windows Search.
    In case you don’t already know, you could stop these services by bringing up the Services window on your machine: Go to Start –> Run and enter “services.msc”. Now double-click on the services of your choice, say Superfetch, hit the Stop button and click “OK”. Once you are done with the build, hit “Start” to turn the service back on.

Additional Useful Information


  • If you have Process Monitor (from SysInternals), you can set it up to monitor the directory to see which applications are reading/writing to it. Perhaps, a 3rd party application could be holding a lock on the files causing the compiler, linker or another VS tool to be waiting for it. You may download Process Monitor from http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx if you don’t already have it on your machine.
  • You may try disabling the Intellisense parser, especially if the memory on your machine is pretty low. On an experiment I conducted on a machine with 1GB RAM, I noticed about a 6% reduction in build time with a medium sized project. Or you may reduce the number of out-of-proc IntelliSense parsers from 2 (default). Check this blog out for detailed information regarding Intellisense options, and for figuring out what you really need and don’t need.
  • The development profile chosen may cause a variation in build time.
    If you want to try changing your current profile (say, to C++): open up the Tools menu, select Import and Export Settings, choose Reset all settings and hit Next; in the next page, select No, just reset settings, overwriting my current settings and hit Next; choose Visual C++ Development Settings and click on Finish.

Visual Studio default settings

On a side note, there is also a way to reset VS to the Out-of-Box state. Read the content under “/resetuserdata Switch” here. While "devenv /resetuserdata" does set Visual Studio 2010 configuration files to default, it does not reset the user configuration, which may have been modified by the user with specific settings. To do this, open up the VS 2010 command prompt and type in the command “del %LOCALAPPDATA%\Microsoft\MSBuild\v4.0\Microsoft.Cpp.*.user.props". This will remove the files containing the user specific settings, and these files will be recreated using Visual Studio 2010 defaults the next time that VS is started.

  • Finally, here is a way you could get fast builds on a multiprocessor computer by taking advantage of build parallelism.

 

II – Common Scenarios/Issues that Would Increase Build Time

  1. In VS 2010, the compiler is set to generate PDBs by default in the Release configuration
    In VS 2010, the default value of Debug Information Format is “Program Database (/Zi)” while in VS 2008, it was “Disabled”.
    In most of the wizard generated projects in VS 2008, “/Zi” was forced and the wizards were overwriting these values in the projects. So, in VS 2010, we made this the default and the wizards no longer force the projects to overwrite this property.
    This could create dissimilarity in property values between VS 2010 and the version converted from. It can surface up only in two scenarios:
    1. If you are upgrading from VC6.
    2. If you are upgrading from a project where someone had explicitly chosen “<inherit from parent or project defaults>”.

    For a little background on why this is happening, VS does not create an entry in the project file whenever the property takes a default value.

    Tip for identifying defaults: Default property values appear unBolded in the project’s Property Pages.

    Generating PDBs increases build time. Therefore, if your situation does not require generating PDBs, you could bring up the Property Pages of the project and disable Debug Information Format by clearing the value of the field corresponding to that property.

    Debug information format

    To step it up a notch and further reduce build times, you may also disable the linker from generating PDBs: Open up the Property Pages, go to Configuration Properties –> Linker –> Debugging and set Generate Debug Info to “No”.

    Generate debug info

  2. In VS 2008 IDE, /O2 did not get passed to the compiler when not explicitly set in the Property Pages
    In VS 2008, although “/O2” was the default compiler Optimization option under Release configuration, say the user did not go and explicitly mention that he wanted the compiler to optimize as “/O2”, then the switch would not get passed to the compiler (since like mentioned above, an entry was not getting created in the project file).
    So how does this affect VS 2010 build time? After converting your project to VS 2010, since this issue has been fixed, your project will build with “/O2” on (as desired) causing the build to take longer (in most cases), since the compiler needs to perform an additional task of optimizing the code.
    So if your priority is to have faster build time and you don’t really care about the optimization, just disable it: open up the Property Pages, go to Configuration Properties –> C/C++ –> Optimization and change the value of the “Optimization” property to “Disabled (/Od)”.

    Optimization property

    This should be implied but I’ll mention it anyways: Due to this issue, say you want to gain a meaningful understanding of the build time slowdown (if at all) in VS 2010, you could do so by eliminating the differences between the VS 2008 and VS 2010 builds. For this, enable “/O2” in VS 2008 and do a build before comparing it with the VS2010 build time. This applies to the “/Zi” case mentioned above as well (Enable “/Zi” in VS 2008 and build).

    Note: Optimization settings make your application run faster on user machines. By turning off optimizations, you are making a conscious choice of accepting that your application will be slower.
    Here is a post on code generation improvements made in Visual Studio 2010 to improve performance.

     
  3. In VS 2010, an MFC app that links to its libraries statically shows reduced build performance
    In VS 2010, there has been a substantial growth of the MFC static libraries, which transforms into additional work for the linker while building (The size of the generated binary is also significantly larger as a result).
    It is actually caused by a new feature added, allowing MFC controls to be added to a dialog. This in turn required that the DLGINIT handler have knowledge about the MFC controls and this pulls in all the additional portions of the library. We see the benefit of the feature as outweighing the drawback of the additional size of a static EXE, so we do not want to pull the feature. We do, however, plan to address this issue in some way in a future release of Visual Studio.
    To work around this issue, consider linking to the MFC libraries dynamically. To do this, open up the Property Pages of the project, go to Configuration Properties --> General and change the "Use of MFC" property to "Use MFC in a Shared DLL".

    Use MFC in a Shared DLL

    Microsoft recommends dynamic linking anyways. This allows users to immediately take advantage of any future security patches that may apply to these libraries without the vendor having to rebuild/reship their application.

 

III – Measuring Build Performance

This section contains suggestions for you to reliably measure the performance if you choose to do a Rebuild (Build -> Rebuild Solution).

A - Initial machine setup

  1. Reboot the machine.
  2. Open a VS command prompt in Administrator mode; run ngen executeQueuedItems and wait for it to complete so the ngen cache will be cleared, just in case.
  3. Start VS IDE and apply the Visual C++ Development Profile.
  4. Load the solution.
  5. Do a Rebuild Solution.

B - Measuring Cold Timing

  1. Shutdown VS.
  2. Open up the Task Manager and kill all instances of MSBuild.exe (for VS 2010 builds) and mspdbsrv.exe.
  3. Start VS, Load the Solution, and wait for the CPU to become idle.
  4. Execute the following steps:
    1. Choose Rebuild Solution.
    2. Immediately, start a timer (like the old times, a stopwatch should still work).
    3. Wait for the build to complete and for the status bar to display a message saying “Rebuild All Succeeded”.
    4. Immediately stop the timer.

C - Gathering Warm Timings

  1. Repeat step 4 in the previous section three times.
  2. Report the average of the three timings collected. If any reading has a lot of variation, redo the build to gain confidence. Or you could perform more number of builds and eliminate the highs and lows.

Note: When you measure the time for the Rebuild Solution scenario, keep in mind that this involves the additional overhead of cleaning up files that were generated during a prior build. For consistency purposes, choose from one of the options listed below:

  • Do a initial build to generate the files, and start gathering the numbers from the next re-build (as explained above); or
  • Do a Clean before every Rebuild, so no files will have to be deleted during the build.

 
IV – Conducting Investigations

A build includes various pieces like MSBuild, IDE, Tools, Tasks ... Listed below are four steps to take to narrow down a slowdown to its root cause:

  1. Find out if it is a Compiler/Linker Tools regression
    The most helpful place to start is with a log of compile/link times by passing the timing switches during build: /Bt for the compiler and /time for the linker.  The question you are really trying to answer is what component slowed down and compiling/linking what thing is the cause.
    Just in case you are wondering how to pass these switches, there are two ways:
    1. Open up the Property Pages of each project (you could multi-select the projects) and go to C/C++ or Linker --> Command Line and add the switch under Additional Options.
    2. Open up a command prompt, set _CL_=/Bt and _LINK_=/time and launch the VS IDE from the command prompt using devenv.exe.
    This blog contains additional information regarding the use of these timing switches.
    Use this information gathered to create a table like:

    Project File Front-end Time
    (c1.dll + c1xx.dll)
    Back-end Time
    (c2.dll)
    Link Time
    (Final: Total Time)
    1 1 X1 Y1 Z1
    2 2 X2 Y2 Z2

    The best way to go about this is to create a script to parse the logs into the format shown above.
    You can then merge these together in (say) Excel for two to three different runs to compare the times for each file/project and see where the big diffs are.  One of two things will happen:
            1. Slowdown is across the board
            2. One or a few things will have regressed greatly.

    In the first case, you may create a repro by extracting any small file or project from the solution. While in the second case, extract a repro of just the particular file or project causing the slowdown.
    You should verify that you can reproduce the slowdown by just compiling or linking the extracted project, ideally by just running the actual command like cl.exe ... or link.exe … You may then use this repro to report the issue to us (There is a section on Reporting Bugs below).
    If you did see a wall clock regression for an entire build but no cl/link tools time slowdown, you have thereby eliminated the cl and linker, and can now proceed investigating the other tools (see Step 2).

  2. If no regression is identified in Step 1, look for regression in Other Tools
    As there are no similar hidden switches for the other tools (Lib, RC, Mt, BSCMake, Resgen, CustomBuild ...), we can use the timing information output by MSBuild to isolate/investigate the regression. Change the build log verbosity to Diagnostic in order for MSBuild to print out these times. But these times include the additional overhead of invoking the tool, etc.
    To view these times, for each project in the solution, find the Task Performance Summary towards the end of the build output. It would look like this:

    Task Performance Summary:
    0 ms SetEnv 4 calls
    0 ms Touch 2 calls
    0 ms ReadLinesFromFile 1 calls
    0 ms Message 3 calls
    0 ms Delete 2 calls
    0 ms AssignCulture 1 calls
    0 ms MakeDir 10 calls
    0 ms WriteLinesToFile 1 calls
    0 ms AssignTargetPath 5 calls
    359 ms RC 1 calls
    937 ms Mt 1 calls
    3843 ms CL 2 calls
    7061 ms Link 1 calls

    Although you were done analyzing the compiler and linker times in Step 1, you might want to consider looking into their Task times reported here as well. If you see any regression, it would mean that the slowdown is stemming from the additional MSBuild overhead sub-tasks and not from the tool. You can then measure this exact time by subtracting the Task time from its respective tool time (calculated in Step 1; note, build log verbosity should have been set to Diagnostic in this case).
    Again, figure out if the slowdown is across the board or in a particular project. If you don't see a regression in any of these tools, it should most probably be a regression in MSBuild itself. However before concluding that, eliminate the IDE factor by performing Step 3.

    Tip: There is a way to cross check if the slowdown you are seeing is being caused by VS 2010 tools (cl, link, mt, etc.) in particular. For this, you will need to have VS 2008 installed on the same machine.
    Open up the Property Pages, go to Configuration Properties –> General, change the Platform Toolset to “v90”, and do a build.

    Platform toolset

    If you see an improvement in build time, you’ve just narrowed down the issue to VS 2010 tools.

     
  3. If no regression is identified through Steps 1 and 2, test whether it is from the IDE
    The next thing you could do is test whether the slowdown is being caused by the IDE. You may determine this by building with MSBuild over the command line and eliminating the IDE completely. If the slowdown still exists, you know for sure that it is not being caused by the VS 2010 IDE.
    For details on how to build with MSBuild over the command line, read this blog.
     
  4. Collect ETW Traces
    ETW stands for Event Tracing for Windows. A tool was released recently that automates collecting these traces. You may find it by clicking here.
    On installation of the tool, you will see a user guide document; follow the instructions in there. Essentially you will just need to start the Profiler, run your scenario and then stop the profiler. Once you have the ETW trace, send it over to us for analysis.

 

V – Reporting Bugs

If after going through the steps above, you still think that there is a performance issue that needs our attention, feel free to get in touch with us through forums if you need some clarification, or open a Connect bug if you would like something to be considered for fixing in a future release.

When getting in touch with us, please be as detailed as possible providing relevant information from the sections above. Provide all variability information regarding your machine configuration (type of processor, number of cores, amount of installed memory, free disk space, etc.)

Attach the project sources (if it does not violate any licensing terms). If there are license restrictions, try removing/obfuscating any IP and send it to us. If that is not feasible either, consider sending us just the solution (.sln) and project (.dsp/.vcproj & .vcxproj) files. Include Diagnostic logs (see section I.2; build output/log file verbosity setting should be changed to Diagnostic) with timing information enabled (/Bt and /time), along with ETW Traces.

Be detailed about the steps you took in narrowing down the issue and the percentage of slowdown you are observing after eliminating any disparities between the two builds that you are comparing. This will help speed up our investigations and enable us to respond to you quicker. Feel free to also provide any subjective observations you made while doing the exercise.

 

I would like to thank you all from my end for using Visual Studio 2010 and being patient enough to read this blog. If you have any questions/concerns/comments, throw them at me and I’ll be happy to respond to them.

 

Renin John
Visual C++ Project and Build team

  • Great post!

  • Very helpful, thanks!

  • What impact do SuperFetch have on build performance ? Is it because of the overhead introduced by the monitoring of each compiler process started ?

  • This is all interesting stuff.  

    Unfortunately the current VS2010 seems to ignore the /Gm switch (Enable Minimal Rebuild) and always rebuild all for my large C++ project which is a real productivity killer and certainly a bug that should have been fixed before release.  This completely negates any underlying improvement in build time.

    Surprisingly Edit & Continue actually works OK without sparking a full rebuild.

    Hopefully Sp1 will fix this issue??

  • I was shocked to read this:

    "In VS 2008 IDE, /O2 did not get passed to the compiler when not explicitly set in the Property Pages

    In VS 2008, although “/O2” was the default compiler Optimization option under Release configuration, say the user did not go and explicitly mention that he wanted the compiler to optimize as “/O2”, then the switch would not get passed to the compiler (since like mentioned above, an entry was not getting created in the project file). "

    A bit of searching yielded forum threads like this one:

    social.msdn.microsoft.com/.../3108af8e-14f9-4cae-9631-981e35263fc3

    It's staggering this wasn't patched as a matter of urgency. The problem isn't even mentioned in the online help pages for the /O2 compiler switch (VS2008 version), despite the point of online help being that it can be updated when issues come to light. IMO a hotfix for this should be issued even now, and pushed through Microsoft Update as a high-priority one!

  • re: MFC static link release size - to get around the problem you can try this: http://bit.ly/dEhcwV

  • @ John, David: Thanks!

    @ Rolf: While Superfetch is known to increase the performance of the machine by making efficient use of memory, we suggested you turn it off as it could cause variability between different measurements. Performance measurements should be taken in a controlled environment. The service itself would add only a small overhead if at all.

    @ Rob: Could you please provide me more information, or maybe create a simplified repro and send it to me? My email is reninj [at] microsoft [dot] com.

    By the way, we have a hotfix available for VS 2010 to enable Managed Incremental Build (support.microsoft.com/.../982721), and it will also be included as part of SP1. You might want to try it out (if applicable) and check to see if the build time shows improvement.

  • Renin: your tips prompted me to look at the Build with max Verbosity.  I noticed that it was the <project>.pch file that was triggering the recompilation of all modules.  The .pch was being recreated at the end of the build.  This was because the project configuration C/C++ Precompiled Headers was Create (/Yc).  I changed to Use (/Yu) and all was good.  I guess that the VS2010 conversion must have changed this switch but otherwise not a bug.  Thanks for your interest.

    Rob

  • @ Andrew: We understand we made an inadvertent change that caused this issue. We will make sure that this is clearly communicated to our support and forum members. We believe that the workaround provided (which is to change the optimization from /O2 to /O1 and back to /O2) will assist you in making the application function as desired.

    Thanks for your passion using our software and we will strive to rectify this in an appropriate fashion.

    @ Rob: I am glad you were able to identify and fix the build issue.

    Regarding /Yu changing to /Yc during conversion, I haven't heard of this issue and was not able to repro it either. If possible, please provide me with a repro so I can investigate the issue further.

  • Thanks for the specific guidance!  

    Do you have any way for us to get more granularity in the /time output? For example,  here's a typical link but I have no idea how to interpret these numbers or how to dig into it further to see what I should move/trim/etc.. to improve our link times.

    1>  LibDef: Total time = 0.733s

    1>    MD Merge: Total time = 24.929s

    1>    Generate Transitions: Total time = 0.562s

    1>    MD Finalize: Total time = 2.824s

    1>  Pass 1: Interval #1, time = 117.812s

    1>    Wait PDB close: Total time = 3.510s

    1>  Pass 2: Interval #2, time = 71.293s

    1>  Final: Total time = 189.105s

  • Hi,  i'm studying computer-science, i'm working on the Microsoft Visual Studio 2010, and once i installed it, i wrote the program i had to write, but it gave me that the project is out of date, and i couldn't do anything more... so i'm lost, and i have an exam soon and  i need help please about what should i do. Should i re-install? or that doesn't work?

    Thank you.

  • Hi Cynthia, if I'm not wrong, the message you're getting is something like "These projects are out-of-date? <List follows> Would you like to build them? Yes/No/Cancel"

    If that were the case, just click on Yes. If we're talking about the same window, you may also check a box labeled "Do not show this dialog again" and it won't prompt anymore.

  • Hi Diego, yes it's like that, but when click yes in the end, i get that "the system cannot find the file specified" and when i open the folder of my projects, and i click on some project, there must be a folder called debug, but there is not. In fact i think the problem is somewhere with the debug, because when i write a project,and i press to start the debug, i can't even see the errors i have , and i tried to re-instait it, but that didn't work. and i'm getting desperate... Thank you for your help, i really appreciate it.

Page 1 of 1 (13 items)