Multi-processor builds in Orcas

Multi-processor builds in Orcas

  • Comments 18

Hey again, everyone.  I’m Peter-Michael, a program manager with the VC++ compiler team.  Continuing with the Orcas feature parade, I'd like to talk about the compiler's team contribution to the product.  Back in June, Jonathan Caves posted about the compiler team’s plans for Orcas.  To recap, the compiler team is using this product cycle to get a head start on our next-generation compiler, the fruits of which we plan on delivering to you beginning with Orcas+1.  In the mean time, we have a smaller subset of the development team working on maintaining the current compiler until that glorious time comes where we can make the big switch over to our new technology.

In that time, we’ve fixed around 180 bugs in the compiler, 71 of those coming directly from MSDN connect feedback.  We’ve continued the standards conformance work that we undertook in VC 8 (as described by Andy Rich in his vcblog post) and fixed several outstanding bugs regarding the mixing of friend declarations with templates.  For example, the following code

template<class T> class A {

      friend void f(const A& a) { }

};

 

void bar(const A<int>& a) {

      f(a);

}

gave an error (cannot find declaration of f) in VC 8 when the instantiation of the template A should have made f(a) a valid call.  In Orcas, we’ve ensured that this and other friend-template and friend name-lookup issues have been resolved.

In addition to maintenance work, we found time to implement one feature that we hope you’ll find useful.  Build times always prove to be a major pain point for our customers, so we have done our best to try to provide technologies to mitigate the cost of building large projects such as the managed incremental build feature (formally called asmmeta) that Curt Carpenter recently blogged about.  With the growing presence of multi-core computers in both the developer and consumer space, exploiting that added power can help reduce build times significantly.  To that end, we’ve added a new switch to the compiler, /MP, to enable multi-process building at the source file level.

/MP works by exploiting the fact that translation units (a source file coupled with its includes) can be compiled independently of each other (up to link time where all object files need to be present).  Since we can compile each translation unit independently, we can parallelize the compilation by spawning multiple processes to handle a batch of source files.  This is precisely what /MP does; when you issue /MPn to the compiler, it spawns up to n processes to handle the source files passed to it.  The compiler is smart enough to spawn only as many processes as necessary, so if you specify three source files on the command-line and invoke the compiler with /MP4, then the compiler only spawns 3 processes.  By default, /MP takes n to be the number of effective processors on the machine.

The history of this switch is rather interesting to boot.  This switch was originally an intern project way back in Everett (VS 2003) but due to technical difficulties and resourcing issues, it never saw the light of day.  Some of our VC++ devs use the switch to build the compiler and see a 20-30% difference in time on a dual-proc machine.  Recently, we gave the switch to a customer in order to help with their build times and they too saw a large performance increase when they used /MP coupled with our current project build parallelization technology, on the order of a  30% gain.  After seeing this data and positive feedback, we decided that it would be prudent if we sat down and polished the switch now and get it out in Orcas.

You can preview /MP and the rest of VC++’s feature set in the March Visual Studio CTP, available as a Virtual PC Image or self-extracting install.  We hope that you enjoy your decreased build times!

Peter-Michael Osera – VC++ Compiler Team

  • > You can preview /MP and the rest of VC++’s feature set in the

    > March Visual Studio CTP, available as a Virtual PC Image or self-extracting install.

    If you want us to preview the /MP switch then you probably need to deliver WMWare images. Virtual PC does not support multiple processors for guest OSes if I do not miss something.

    Moreover if you are really interested in the final result and want us to test Orcas on real projects you probably should not change the format of project files. Many of us do use software configuration management tools in our everyday work.

    > We hope that you enjoy your decreased build times!

    We do enjoy them with IncrediBuild  for many years already.

    But anyway Orcal looks very promising. Some annoyed VS2005 problems (like comatose Pending Checkings pane) are gone.

  • I use SCons as a build tool and it is quite handy to be able to say scons -jN so that it runs N build steps in parallel, keeping the order correct.

    Reliance on the IDE requires /MP

  • Most compiler switches are exposed as project properties, but this makes more sense as a machine-specific VS option. Will this be stored in the .vcproj, the .user/.suo, or the main VS options? It would be a pain to be stored in the project properties in a multi-user source controlled environment.

    Wll this default to on - otherwise people might not know it's there and miss out?

  • Any chance the linker will be multi-processor aware someday?

  • Hello,

    Are you sure the download site for the image gives the right admin credentials?

    I've tried with various (P2ssw0rd, p2ssw0rd, P@ssw0rd, p@ssw0rd) but been unable to get in.

    Any hint there?

    Thanks.

  • > Any chance the linker will be multi-processor aware someday?

    I doubt it may happen in the nearest future. Today the situation goes from bad to worse -- the modern vc7/8 linkers are twice as slow in comparison with vc6 one.

    But they at least are capable to handle pdb files that are more then 64Mb in size. For some projects it is vital because the mentioned IncreadiBuild has to place debug info into obj files to be able to compile sources in parallel. And if it exceeds 64Mb the vc6 linker chokes.

    So, IMO the situation with linkers is the real pain.

  • you annouce 30% of speed gain on a dual core, do you have numbers for a quad core ?

  • Andrew: the final location of this setting is still under discussion.  There are pros and cons to user vs. solution vs. project.

    PWC: The Phoenix project (next-generation C++ compiler back-end) is working on adding parallelism to the back-end, although I don't know to what degree that will affect the linker vs. just the code generator.  To me, a better solution is to modernize the C++ build model and get rid of the linker altogether, but this is a topic for a different day...  :)

  • Hello, everybody!

    I have installed march CTP in my computer and my computer has dual-core CPU.

    But hoooooow can I turn /MP on in the IDE?

    I can't find it!

  • Peter,

    It sounds like a worthy addition, how about compilation using different computers.  In a shop of 5+ programmers, not all computers will be doing compilation at the same time, if we can tap into unused cycles on the network, it will be even better.

    4-5 years back when I was working on QNX 4.2, the C++ compiler was already doing out of the box!

  • Composition: As Steve mentioned, we are still figuring out the final location of the switch in the IDE.  You can add /MP to your command-line settings for the project by going to Project->Properties->Configuration Properties->C/C++->Command Line.

    Quoc: We don't have any plans to be adding network-distributed builds, but as Michael mentioned above, Incredibuild is a 3rd-party tool that can help out in the scenario that you described.

  • To allow customers to evaluate and plan for our upcoming "Visual Studio Orcas" release (and to highlight

  • 轉貼自http://blogs.msdn.com/vcblog/archive/2007/04/10/visual-c-orcas-feature-specifications-online.aspx...

  • Hi, I don't quite understand from your post is whether /MP is supposed to work on compiler (cl.exe) or msbuild?

    Can you please clarify?

  • It doesn't make sense for /MP to be a project setting. For example, it's great for that luck bastard who has the dual quad core box who runs /MP16, but it sucks for you on your laptop!

    The rule of thumb for GCC that I've always been told is to set twice the number of concurrent builds as you have processors. So on my quad core, I would use /MP8. On my laptop, /MP2.

    Multiprocessing is ment to exploit the capabilities of the machine that it is being compiled on. Therefore, that option should be set per machine, rather then for everyone who uses the project.

    Just my 2 cents!

    ... but whatever you do, don't take /MP out!!

Page 1 of 2 (18 items) 12