_ITERATOR_DEBUG_LEVEL - Advanced STL, Part 3

_ITERATOR_DEBUG_LEVEL - Advanced STL, Part 3

  • Comments 10

In Part 3 of my video lecture series exploring the Standard Template Library's implementation, I explain how our powerful correctness checks in debug mode work.  In VC10, they're controlled by the macro _ITERATOR_DEBUG_LEVEL, which supersedes VC8 and VC9's more confusing _SECURE_SCL and _HAS_ITERATOR_DEBUGGING macros.  Additionally, VC10's #pragma detect_mismatch allows the linker to detect badness that previously would have caused incomprehensible crashes.  Finally, I demonstrate the undocumented and unsupported but exceedingly awesome compiler option /d1reportSingleClassLayout which prints an ASCII art diagram of a class's representation.

 

This advanced series assumes that you're familiar with C++ and the STL's interface, but not the STL's implementation.  If you haven't used the STL extensively yet, I recommend watching my introductory series.  For reference, here are all of the links:

 

[STL Introduction]

Part 1 (sequence containers)

Part 2 (associative containers)

Part 3 (smart pointers)

Part 4 (Nurikabe solver) - see Wikipedia's article and my updated source code

Part 5 (Nurikabe solver, continued)

Part 6 (algorithms and functors)

Part 7 (algorithms and functors, continued)

Part 8 (regular expressions)

Part 9 (rvalue references)

Part 10 (type traits)

 

[Advanced STL]

Part 1 (shared_ptr - type erasure)

Part 2 (equal()/copy() - algorithm optimizations)

Part 3 (_ITERATOR_DEBUG_LEVEL, #pragma detect_mismatch, and /d1reportSingleClassLayout)

 

Stephan T. Lavavej

Visual C++ Libraries Developer

  • Could someone update the MSDN documentation? The pages and contents tree are all still referring to _SECURE_SCL etc.

    msdn.microsoft.com/.../aa985965.aspx

    Also, on that page someone's produced some "community content" pointing out a discrepancy in the stated default values. Which is right? Or is that not applicable in VS2010 anyway, what with the change in macros?

  • Thank you for the heads up, Andrew. I'm escalating this issue to the documentation team.

  • I hate to be so cynical, but is this in the same "escalation" as when we notified you quite a few months ago,  that all the Direct3D 9 docs are missing entirely from VS2010's Help system? They're still not there.

    Maybe this would make an interesting guest post at some point in the future. Get someone to explain how documentation is added to MSDN (or, implicitly, how such key pieces of documentation as major Microsoft libraries, and key compiler macros, can just *get lost* along the way)

    As a naive outsider, I'd have assumed that there was someone, somewhere, who was responsible for these things.

  • If you attempt to turn off the iterator debugging the PDB file is no longer correct.

    This also cause problems with purifyplus which appears to stop working. You want to turn off iterator debugging as it affects performance measurements.

    See: connect.microsoft.com/.../vs2010-debugger-watching-std-string-with-iterator-debug-level-0-does-not-work

    This is a big show stopper for us.

  • Damian: That bug is currently assigned to me. I've identified the problem - it's nasty and subtle, and unfortunately I wasn't aware of it during VC10's development. I've prototyped a fix, but I still need to polish it up and check it in. Note that because the fix requires invasive brain surgery on our facet cleanup machinery, it will be VC11-only and it will not be available as a VC10 hotfix.

    > You want to turn off iterator debugging as it affects performance measurements.

    This is incorrect. As I explained in the video, IDL defaults to 0 in release mode, which is what you should be running perf tests on. IDL defaults to 2 in debug mode, and you shouldn't be running perf tests in debug mode - even if you're compiling *optimized* debug mode (which would be very strange), you'd still be linking against an unoptimized CRT, invalidating any perf numbers.

  • Related to out-of-date docs, what's the status of _SECURE_SCL_THROWS? I know it's deprecated in VC10 (but at least seemed like it nominally still worked in a simple test), is there a replacement available? And if so, is it just going to be through _invalid_parameter_handler?

    I know certain objects (I think std::string (and I'm guessing it's counterpart std::wstring)) didn't respect the flag since the implementation was in msvcp90.dll, but it was generally moderately useful for cases where pdb's are unavailable (and produced more useful information than an _invalid_parameter_handler would).

  • Nothing against STL of course (I assume you're just following company orders), but that whole "hoard everything until the next major release" thing is really getting absurd, and is just hurting your product's value. This isn't the 80's any more. Release often. Do your customers the favor of letting them *use* the improvements you make.

  • Stephen: What is the timeframe for VC11?

    > This is incorrect. As I explained in the video, IDL defaults to 0 in release mode, which is what you should be running perf tests on.

    I was aware of this. We do a release build for perf tests but with debug symbols on.

    purify & purecov uses the pdb to give us the link between the source code (line and function) and the timings.

    I'm not sure how purify and purecov instrument the binaries. All I have been able to discover so far is that a debug build works while a 'purify build' does not (purify build is basically a release build with debug symbols and inlining turned off for purify, inlining turned on for quantify).

    Occasionally we have to debug release mode code. This is usually where the behaviour is different between debug and release builds.

  • Michael> Related to out-of-date docs, what's the status of _SECURE_SCL_THROWS?

    _SECURE_SCL_THROWS was deprecated in VC10, but was otherwise unchanged. It has been completely REMOVED in VC11, and any attempt to enable it will trigger a hard #error.

    Apparently you were using it for diagnostics, but _SECURE_SCL_THROWS was really the most vile abomination ever. Quoting myself from an internal E-mail:

    "_SECURE_SCL_THROWS has been removed from VC11 - I have already checked in this change, and I will not revert it.

    You should choose Option B, remove _SECURE_SCL_THROWS now.  If you're manually enabling _SECURE_SCL (which is disabled by default in VC10 release mode - this is a change from VC8/9), your program is buggy and is misusing the STL (possibly under the control of an attacker), and _SECURE_SCL detects this (it can detect some but not all STL misuse), your program will be immediately terminated.

    What _SECURE_SCL_THROWS did was to replace this immediate termination with a thrown exception.  This was the most horrible thing possible.  The only purpose of _SECURE_SCL is to improve security, but _SECURE_SCL_THROWS weakens security.  When your program is buggy and is experiencing a possible attack, the worst thing in the world that you can do is attempt to recover from it.  That'll execute more code, which can potentially be exploited.  Most ominously, if recovery is successful, an attacker can attempt to exploit your program thousands or millions of times per second, until they get lucky and _SECURE_SCL doesn't detect it."

    > And if so, is it just going to be through _invalid_parameter_handler?

    I believe (but have not tested) that you can customize _SCL_SECURE_INVALID_PARAMETER. However, I would characterize that as untested and unsupported.

    > I know certain objects (I think std::string (and I'm guessing it's counterpart std::wstring))

    > didn't respect the flag since the implementation was in msvcp90.dll

    string/wstring are header-only in VC10.

    > but it was generally moderately useful for cases where pdb's are unavailable

    > (and produced more useful information than an _invalid_parameter_handler would).

    I believe that you can capture a memory dump and then debug it on a machine with PDBs to see the callstack - but I don't know very much about such things.

    Damian Dixon> Stephen: What is the timeframe for VC11?

    I can't talk about release dates, sorry.

    STL> This is incorrect. As I explained in the video, IDL defaults to 0 in release mode, which is what you should be running perf tests on.

    Damian Dixon> I was aware of this. We do a release build for perf tests but with debug symbols on.

    Release mode versus debug mode (i.e. /MT versus /MTd, or /MD versus /MDd), the presence of debugging symbols (e.g. /Zi), and the level of optimizations (e.g. /Od versus /O1 versus /O2) are all orthogonal. _ITERATOR_DEBUG_LEVEL is sensitive to release mode versus debug mode only. If you're profiling optimized release mode with debugging symbols, _ITERATOR_DEBUG_LEVEL will default to 0.

  • > Nothing against STL of course (I assume you're just following company orders),

    > but that whole "hoard everything until the next major release" thing is really

    > getting absurd, and is just hurting your product's value. This isn't the 80's any

    > more. Release often. Do your customers the favor of letting them *use* the

    > improvements you make.

    I couldn't agree more..

    The result of this lazy (yes, lazy) policy is that the supposed fix only fixes part of the problem. This creates (or should I say strengthens) a disconnect between the dev team who think that the bugs are few and far between, because, after all, they have just "fixed" all of them, and us, users, who are being assured over and over that the show-stopper bugs we reported will be taken care of in the next release, only to find out every time that the bugs are still there, mostly intact.

    How many times have we been told that the new version of Visual Studio finally fixes the performance of Intellisense, for example? Just look at the post from Mar 29 on this blog. The dev team evidently thinks that apart from Intellisense not working for C++/CLI, there are no major problems in this area. Yet this is far, far, FAR from being the case.

    Sigh...

Page 1 of 1 (10 items)