STL: Destructor of Bugs

STL: Destructor of Bugs

  • Comments 10

I'm the newest developer on the Visual C++ Libraries team. My true name is Stephan T. Lavavej, but I often use my extremely convenient initials. I moved here at the beginning of this year after working on and finishing Outlook 2007 Search.

 

You know how some people are fans of the Harry Potter novels, standing in line outside a bookstore at midnight in order to be one of the very first to experience a new adventure of wizardry? I'm that way with the International Standard for C++. (My following comments apply to "native" C++. I don't do managed code.) It's a big document, describing the biggest of languages. Its standard library is of moderate size, but with heavy emphasis on general computation, which is far from being the solved problem (in terms of abstractions, not Turing-completeness) that it is often considered to be.

 

There's something about the container-iterator-algorithm-functor structure made possible by C++ and introduced by the Standard Template Library which makes it simultaneously powerful and lightweight. By attacking old problems (how do we generalize containers and algorithms?) in new ways (with templates at compiletime, not inheritance at runtime), the STL pushed bugs - the bane of programmers - to compiletime instead of runtime. Finding errors earlier is always better than finding them later, even if you have to decode unwieldy template error messages (something that the next C++ Standard should help with). Making errors more obvious and quicker to find isn't the only great thing that the STL does, but it's certainly one of the most important.

 

Of course, not all badness can be found at compiletime. The STL has a fairly comprehensive focus on performance, as does all of C++. This means that Standard Library implementations aren't required to perform bounds checking. The structure of the STL makes it harder for properly written programs to commit bounds errors, but not impossible. Also, some forms of checking are inherently incompatible with maximum performance. The STL permits advanced container manipulations, like inserting and erasing ranges of elements into a container, or removing all duplicates from a container, and so forth. Programs can simultaneously maintain iterators into containers, and these iterators can be invalidated when their containers are modified. For example, when enough elements are pushed back into a vector, the vector eventually runs out of capacity and must reallocate its elements into a larger chunk of memory. Iterators which used to point to the elements in their old locations are thereby invalidated. (So are pointers and references.) Performing runtime checking for invalidated iterators imposes a significant performance cost, so Standard Library implementations aren't required to do it.

 

Iterator invalidation isn't a huge, nasty problem. But it's also nice for programmers to be able to have their compiler perform thorough correctness checks. Hence, Visual Studio 2005 provides iterator checking and debugging. Iterator checking (controlled by _SECURE_SCL) is enabled by default in both release and debug mode, and detects bounds errors. Iterator debugging (controlled by _HAS_ITERATOR_DEBUGGING) is enabled by default in debug mode (it cannot be enabled in release mode), and detects invalidation errors. You don't have to do anything special to use iterator checking and debugging, you just get them for free.

 

However, everything has bugs, including bug-finding machinery. So far I've worked on a couple such bugs. One of them, a regression in Visual Studio 2005 SP1, was triggered by compiling a program in debug mode, with iterator debugging disabled (which is not the default), obtaining an iterator into a container, and destroying the container before the iterator. The reason why this triggers a crash is long and complicated, and figuring out the history was fun. We've fixed this regression in Orcas.

 

Another bug, triggered by less obscure code, involves compiling with _SECURE_SCL enabled but _HAS_ITERATOR_DEBUGGING disabled (the default in release mode, obtainable in debug mode) and swapping two containers. Iterators pointing into those containers should still be valid after the swap, but the iterator checking machinery gets confused, triggering an assertion. We're working on fixing this in Orcas.

 

Bugs in the Standard Library implementation, which hundreds of thousands (probably millions?) of programmers count on to function properly, aren't fun. Here's the good news: we're aware of these two bugs, and many others, thanks to customers who reported them through Microsoft Connect. Our human testers and their automated tests find many bugs, but they'll inevitably miss bugs that programmers worldwide will go on to encounter. You, our customers, have tens (probably hundreds?) of millions of lines of source code, exercising every dark corner of the core language and standard library. If you find a bug, and you're very confident that it's not in your code, create a good test case and submit it to Microsoft Connect. It'll wind its way through our processes, and eventually end up on the desk of a real person like me. You'll even get a response from us when we resolve it. I'm still impressed every time I see it in action.

 

Stephan T. Lavavej

Visual C++ Libraries Developer

 

  • Stephan,

    Great that you are working on this. Speaking of catching errors early, one more "bug" I think the _SECURE_SCL code has is that it allows you to link an invalid set of compilation units together. That is you can link files compiled with and without _SECURE_SCL turned on. It would be nice if the linker forceably stopped you from making this mistake. The worst part is I have demonstrated it is possible the code that passes iterators between such units might run.

  • James,

    That's not exactly a bug, but it certainly is a very sharp, jagged edge that you can cut yourself on.

    The general problem is that the linker does not diagnose all One Definition Rule (ODR) violations. Although not impossible, this is a difficult problem to solve, which is why the Standard specifically permits certain ODR violations to go undiagnosed.

    I would certainly love for the compiler and linker to have a special mode that would catch all ODR violations at build time, but I recognize that that would be difficult to achieve (and would consume resources that could perhaps be put to even better use, like more conformance). In any event, ODR violations can be avoided without extreme effort by properly structuring your code, so we as programmers can cope with this lack of linker checking.

    However, macros that change the functionality of code by being switched on and off are flirting dangerously with the ODR, and the specific problem is that _SECURE_SCL and _HAS_ITERATOR_DEBUGGING both do exactly this. At first glance, this might not seem so bad, since you should already have control over which macros are defined project-wide in your build system. However, separately compiled libraries complicate things - if you've built (for example) Boost with _SECURE_SCL on, which is the default, your project must not turn _SECURE_SCL off. If you're intent on turning _SECURE_SCL off in your project, now you have to re-build Boost accordingly. And depending on the separately compiled library in question, that might be difficult (with Boost, according to my understanding, it can be done, I've just never figured out how).

    What can we learn from this, other than (a) always remember the ODR, and (b) macros are evil (to quote the C++ FAQ Lite)? Well, I've observed something interesting: you are not supposed to mix object files compiled with different /MD, /MDd, /MT, /MTd options. But if you do, you get a LNK4098 warning! The linker has somehow been made to recognize this badness (since at least VC7.1, according to MSDN documentation). A casual glance at the source hasn't shown to me how this works, but it works somehow.

    To be clear, I cannot promise (and you should not expect) that anything will change here. However, I will think about how we trigger this LNK4098 warning, and whether something like that might also be applied to _SECURE_SCL and _HAS_ITERATOR_DEBUGGING, so that it would be harder for silent badness to occur.

    Thanks!

    Stephan T. Lavavej

    Visual C++ Libraries Developer

  • Stephan,

    I would assume the linker just looks at the info in the .obj file (that the compiler flag /Zl controls) to determine which CRT's were specified at compile time. I'm going to take a wild guess that this mechanism isn't going to be extendable in any meanful way to solving the ODR problem. Too bad, seems like that was a special case of a set of ODR-like problems.

    Don't worry, I wasn't holding my breath for a fix :-) We'll be dealing with this for years.

    Too late now, but I would have preferred having yet another set of C++ runtimes to link against that would enforce (through undefined symbols) this flag one way or the other. That doesn't scale but nor does the notion of having switches that are quite this jagged and dangerous.

    Cheers,

    - James

  • Lots of VC++ happening recently (in no particular order): ATL Server is now shared source Marina Polishchuk

  • Lots of VC++ happening recently (in no particular order): ATL Server is now shared source Marina Polishchuk

  • Is there are work around to the _SECURE_SCL swap bug in release mode not involving recompiling the codebase including boost etc using _SECURE_SCL=0, and not rewriting your code not to swap containers?

    /Michel

  • Yes, there are workarounds for the swap bug other than avoiding swapping or disabling _SECURE_SCL.

    (First, I should mention that the swap bug has now been fixed in Orcas! The fix applies to all Standard containers except std::string. Additionally, std::deque was broken even when _SECURE_SCL was disabled; it has also been fixed.)

    References and pointers don't trigger the problem, so using them is a workaround.

    Another workaround is translating iterators to indices and back (for the random-access sequences). That is, if you have an iterator into a vector or deque, figure out what index it points to (index = iter - cont.begin()), swap the containers, and then get an iterator from the index (iter = other.begin() + index).

    If you have further questions, feel free to E-mail me at stl@microsoft.com .

    Thanks!

    Stephan T. Lavavej

    Visual C++ Libraries Developer

  • Stephan - I noted your blog post regarding stl bugs, specifically _SECURE_SCL.  This note is more regarding your comment regarding compiling boost with _SECURE_SCL off.  I figured out how to do this since I was having the same compatability issues myself.  You can _SECURE_SCL off through editing this header:

    \boost_1_33_1\boost\config\compiler\visualc.hpp

    I just added the following to the start of the header.

    #ifdef _SECURE_SCL

    #undef _SECURE_SCL

    #endif

    #ifdef _HAS_ITERATOR_DEBUGGING

    #undef _HAS_ITERATOR_DEBUGGING

    #endif

    #define _SECURE_SCL 0

    #define _HAS_ITERATOR_DEBUGGING 0

    This suits my purpose, though I did get a few build errors in a few libraries that I'm not using.  Hope this was helpful.  Keep up the informative blogging!

    Michael Henson

  • QUOTE:

    First, I should mention that the swap bug has now been fixed in Orcas! The fix applies to all Standard containers except std::string. Additionally, std::deque was broken even when _SECURE_SCL was disabled; it has also been fixed.

    END QUOTE

    Regarding Orcas:

    I have to like with 2005. Can I gets fixed for that compiler?

    Regarding deques:

    You say the deque is broken even when _SECURE_SCL=0. Can you elaborate on this? Are you saying the deque is not usable if swap is called?

    Regarding Service Packs:

    Are any of there issues resolved by SPs? I've got SP1.

  • (Apologies for the gaps in my knowledge of the STL and its inner workings but...)

    If my solution uses a mixture of 3rd-party LIBs and/or OBJs which may well have been built with different values for _SECURE_SCL, how am I to workaround this mixture of _SECURE_SCL=0 and _SECURE_SCL=1??  Do I need to wrap all 3rd-party objects in DLLs?

Page 1 of 1 (10 items)