FreiK's WebLog : Developer Info

How to use RtlUnwindEx

Kevin Frei — Thu, 15 Mar 2007 18:24:00 GMT

Disclaimer: I've never actually written an unwind personality routine, so take what's here with a grain of salt.

A few days back, I spent 30 minutes defending the C++ runtime's exception handling personality routine to the guy that has the less than enviable job of supporting the .Net Framework's exception handling personality routine. The eventual point to be learned is that the documentation for RtlUnwindEx is abysmal, and can cause the VC 8.0 C++ runtime to leak objects left and right.

The problem revolves around the C++ team's decision (which I fully support) to cause code compiled with /EHs or /EHsc flags to only invoke destructors when a C++ exception occurs. This means that if you take a hardware exception, call RaiseException yourself, or call longjmp none of your C++ destructors between the point of exception and the handler will be invoked. I've talked about why this is a good idea, but I'll sum it up with this: If that scenario seems bad to you, then you should be compiling your code with /EHa.

Anyway, back to the issue: Say I'm writing a 'personality routine' which will be inovked by the OS when an exception occurs. If you're writing the kind that requires that your handler initiates the second phase of unwind, you need to call RtlUnwindEx. The documentation, which is arguably bad (it's little more that a function declaration), indicates that the EXCEPTION_RECORD structure to pass in is optional. And that's true: the functions on the stack will still be unwound. But the personality routines for those functions might want to actually know that they're being invoked due to an exception, and not for something like a longjmp. So, if you have an exception record that could be passed to the RtlUnwindEx function, you should. If you don't, another personality routine might assume that there's not an exception to examine, and might decide to do something like NOT invoke the destructors for the object on the stack.

BTW - this bug exists only on x86, and only when you're crossing a managed code -> native code boundary, in particularly weird scenarios. And I hope it's getting fixed :-)

-Kev

AMD64 unwind info gotchas

Kevin Frei — Mon, 12 Mar 2007 20:34:00 GMT

I had a brief e-mail exchange with one of the devs on the optimizer team about a checkin he put up for review. He modified the compiler so that it only aligns the stack for functions that call other functions - that's the typical definition in compiler lingo of a 'leaf function'. My first response was "don't do that - you may still have to align for reasons A, B, & C". To which he responded with a quote from the ABI doc that explicitly says you only have to align the stack if you're calling a function. So I started reading. Turns out, he's right (and so is the doc), but there are some really nasty gotchas involved. When we initially did this stuff, we just lived with the scenario where you might be wasting a bit more stack space...

The theme of the problems revolve around restrictions on encoding information in the unwind data. If you are saving XMM registers and have an unaligned stack pointer, you must use the SAVE_XMM128_FAR opcode because the SAVE_XMM128 opcode will only accept offsets in multiples of 16 bytes. Or I guess you could use MOVUPS instead of MOVAPS to save & restore your XMM register, but I wouldn't recommend that on current hardware. Similarly, you have to use the ALLOC_LARGE descriptor for stack allocation of sizes that aren’t [n * 8 + 8]. ALLOC_LARGE actually has 2 variants - one that multiples by 8, and the other than doesn't. You can use the latter to allocate random amounts of data from the stack. But then if you want to use a frame pointer, you're going to be in a pretty weird spot, because it will have to be unaligned, as well. Unwind data dictates that you can only have a frame pointer that = RSP + 16 * [1-15].

I'll probably come back to this article & add some nice hyperlinks to ABI details in the doc itself, but it's been a while since I blogged anything useful, so I figured I'd just get this out there quickly.

New Job, SP1

Kevin Frei — Tue, 19 Dec 2006 20:42:00 GMT

I recently switched from the lead of the Code Gen & Tools team for Visual C++ to the lead for the Code Gen team for the .Net Framework. I've not used managed code much, but for some classes of code, it sure does look easier to use [I'm working on a goofy GUI app for my personal use in C# that took me forever to do in C++]. There are lots of interesting problems to solve in this space, and a dramatically different set of customer constraints & potential customer scenarios. I have to admit the security concerns for a JIT in the ASP.NET world are a little bit unnerving...

In far more interesting news, VS 2005 SP1 just released. OK, it actually released a few days ago, but my power didn't come back on from our nasty storm Thursday night until Monday, so I was a bit preoccupied. I highly recommend picking this up if you're using the AMD64 compiler. We shipped an awful lot of compiler fixes, along with a few code quality improvements, in this release. You'll notice that prologues and epilogues of complex functions look much different, and that the compiler throughput (the amount of time it takes to compile a function) is a bit better. There are some severe bugs in the PGO scenario {Profile Guided Optimization} that anyone using this feature really needs. Aside from that, I hear that the VS experience is just much more pleasant than VS 2005 RTM. Go forth and Download!

Anyway, I'll try to get some more interesting blogs posts going in the future - work seems to have calmed down quite a bit, since Vista finally shipped.

Speaking in 'public'...

Kevin Frei — Wed, 18 Oct 2006 06:35:00 GMT

I'll be hanging out with the cool kids at the Northwest C++ Users Group tomorrow night. If you're in the area, and want to heckle me, swing by. We'll be in building 40 at 6:30 PM. My talk starts at 7:00 PM. I'm talking about the actual runtime cost of exception handling for x64 and x86 on Windows.

Update:

Check out http://www.nwcpp.org/Meetings/2006/10.html for the slides, and a video of the talk, if you're interested. You can see my thinning hair in back, my marvelous posture, and my always entertaining speaking style.

A more flexible array template [and the evolution of the idea]

Kevin Frei — Tue, 19 Sep 2006 00:17:00 GMT

{Disclaimer - I started with a bunch of code from boost::array - it's a great implementation of the functionality it provides}

While my day job doesn't generally allow me to goof around with advanced C++ features, I still managed to find time to do so after the kids are in bed. Mark & I were talking about a new feature in C99: variable length arrays. They're incredibly useful for certain scenarios, and will, by most compilers I imagine, just get turned into an _alloca call. This led to a discussion of the value of variable sized arrays, and to categorize them. Here's the final result:

Type 1: Fixed size arrays: Easy to do in C, boost::array<T, N> does it for C++ with all the bells & whistles of an STL container [tr1 has something awfully similar]
Type 2: Variable length arrays: Hard to do in C, std::vector<T> does it for C++
Type 3: Variable sized arrays at construction time: Easy to do in C (malloc or _alloca always produce these), doesn't exist currently in either boost or STL

When we started comparing these things, we came to the conclusion that C99 style variable length arrays would be very useful. Using __forceinline and _alloca, you can produce an implementation that works for any scenario that stays in a single function scope. As soon as you start working across scopes, or try to implement something as a global, it falls to pieces. It also fails miserably if the compiler doesn't actually inline forceinline functions [which is the case for -Od, and anything with -Ob0]. And I don't have a very reliable way of detecting this scenario :-(

Ignoring the weaknesses, let's assume that the programmer is very intelligent and all-knowing, and will never accidentally abuse these routines. Here's an initial implementation (just the part that matters) that supports stack-based allocation of both Type 1 and Type 3 arrays:

public:
    array(std::size_t n = N) { BOOST_ASSERT(n == N); }
};

template<typename T>
class array<T, 0> : public impl::_common_array<T, array<T,0> > {
protected:
    friend impl::_common_array<T, array<T,0> >;
    T *elems;
public:
    __forceinline array(std::size_t s) : sz(s), elems((T*)_alloca(sizeof(T)*s)
    {
        for (size_t i = 0; i < s; i++)
            T *temp = new(&elems[i]) T;
    }
    ~array()
    {
        for (size_t i = 0; i < sz; i++)
            elems[i].~T();
    }
};

But I don't like allowing a programmer to shoot themselves in the foot accidentally (on purpose is fine :-)). First, I added some code so that we don't try stack-allocation for DEBUG code, since the inliner is rarely enabled for that scenario. Next I promoted a warning to an error so that if the routine ever doesn't get inlined, you'll fail. Finally, I declared a private operator new, so that you can't construct one of these things on the heap and really screw up the world:

template<class T>
class array<T, 0> : public impl::_common_array<T, array<T,0> > {
private:
    void *operator new(std::size_t sz);
protected:
    friend impl::_common_array<T, array<T,0> >;
    T *elems;
public:
#if defined(_DEBUG) || defined(DEBUG)

#pragma message("Using lame implementation")

    array(std::size_t sz) : sz(sz), elems(new T[sz])
    {}
    ~array() {delete[] elems;}

#elif !defined(NODEBUG) && !defined(NDEBUG) && !defined(_NODEBUG)

#pragma message("Be sure that your optimized build is NOT using -Ob0 - this completely disables inlining")
#pragma error("Please define DEBUG or _DEBUG for your debug build, and NODEBUG, NDEBUG or _NODEBUG for your optimizing build, otherwise, this code won't work!")

#else

#pragma message("Using good implementation")
#pragma warning(error:4714)
#pragma optimize("b1", on)
#pragma auto_inline(on)

    // If this thing doesn't get inlined, the code won't work - you'll need to use the previous blob of code instead
    __forceinline array(std::size_t sz) : sz(sz), elems((T*)_alloca(sizeof(T)*sz))
    {
        for (size_t i = 0; i < sz; i++)
        {
            T* temp = new(&elems[i]) T;
        }
    }
    ~array()
    {
        for (size_t i = 0; i < sz; i++)
            elems[i].~T();
    }
};

But I'd really like to allow these things to be a bit more general purpose! Fact is, a tricky optimizer could turn a malloc/free into an alloca, and a fixed-size alloca into a zero-cost part of the stack frame adjustment. But getting the compiler to decide that a variable sized malloc should be turned into an alloca seems error-prone. What if I want an array of 4Mb?
There goes my stack - poof - in one fell swoop.

So, now, I'm headed to some sort of policy based memory allocation where the programmer could indicate that stack allocation should be used.

It's still unsafe, but not automatically so. The programmer is the one who should understand the implications of the malloc/free -> alloca conversion. If there is no allocated memory, we'll just new & delete the memory manually. So, instead of auto-allocating memory on the stack, we'll allow the programmer to allocate memory that they wish to keep track of manually, or, if no memory is provided, we'll allocate it from the heap, and free it when the object is destroyed.

namespace kbf
{
    namespace impl
    {
        template <typename T, class CHILD>
        class _common_array
        {
        protected:
            __forceinline CHILD *downcast() { return (CHILD *)this; }
            __forceinline const CHILD *downcast() const { return (const CHILD *)this; }
        public:
            // type definitions
            typedef T              value_type;
            typedef T*             iterator;
            typedef const T*       const_iterator;
            typedef T&             reference;
            typedef const T&       const_reference;
            typedef std::size_t    size_type;
            typedef std::ptrdiff_t difference_type;
            typedef std::reverse_iterator<iterator> reverse_iterator;
            typedef std::reverse_iterator<const_iterator> const_reverse_iterator;

            // iterator support
            iterator                begin()        { return downcast()->elems; }
            const_iterator          begin() const { return downcast()->elems; }
            iterator                end()          { return downcast()->elems+downcast()->size(); }
            const_iterator          end()    const { return downcast()->elems+downcast()->size(); }
            reverse_iterator        rbegin()       { return reverse_iterator(downcast()->end()); }
            const_reverse_iterator rbegin() const { return const_reverse_iterator(downcast()->end()); }
            reverse_iterator        rend()         { return reverse_iterator(begin()); }
            const_reverse_iterator rend()   const { return const_reverse_iterator(begin()); }

            // at() with range check
            reference       at(size_type i)       { rangecheck(i); return downcast()->elems[i]; }
            const_reference at(size_type i) const { rangecheck(i); return downcast()->elems[i]; }

            // front() and back()
            reference       front()       { return c_array()[0]; }
            const_reference front() const { return data()[0]; }
            reference       back()        { return c_array()[downcast()->size()-1]; }
            const_reference back() const { return data()[downcast()->size()-1]; }

// direct access to data (read-only)
const T* data() const { return downcast()->elems; }

// use array as C array (direct read/write access to data)
T* c_array() { return downcast()->elems; }

// assign one value to all elements
void assign (const T& value) { std::fill_n(begin(),downcast()->size(),value); }

            // check range (may be private because it is static)
            void rangecheck (size_type i) const {
                if (i >= downcast()->size()) {
                    throw std::range_error("array<>: index out of range");
                }
            }

            // operator[]
            reference operator[](size_type i)
            {
                BOOST_ASSERT( i < downcast()->size() && "out of range" );
                return c_array()[i];
            }

            const_reference operator[](size_type i) const
            {
                BOOST_ASSERT( i < downcast()->size() && "out of range" );
                return data()[i];
            }
        };
    }

    template<typename T, std::size_t N = 0>
    class array : public impl::_common_array<T, array<T,N> > {
    protected:
        friend impl::_common_array<T, array<T,N> >;
        T elems[N];    // fixed-size array of elements of type T
    public:
        array(std::size_t n = N) { BOOST_ASSERT(n == N); }
        // size is constant
        static size_type size() { return N; }
        static bool empty() { return false; }
        static size_type max_size() { return N; }

        // swap (note: linear complexity)
        void swap (array<T,N>& y)
        {
            std::swap_ranges(begin(),end(),y.begin());
        }

        // assignment with type conversion
        template <typename T2>
        array<T,N>& operator= (const array<T2,N>& rhs)
        {
            std::copy(rhs.begin(),rhs.end(), begin());
            return *this;
        }

};

    template<typename T>
    class array<T, 0> : public impl::_common_array<T, array<T,0> > {
        // TODO: This needs to be noncopyable, probably, or deep-copyable...
    private:
        void *operator new(size_t);
    protected:
        friend impl::_common_array<T, array<T,0> >;

        std::size_t sz:sizeof(std::size_t) * 8 - 1;
        unsigned int releaseMem : 1;
        T *elems;

    public:
        array(std::size_t s, void *memory = NULL) : sz(s), releaseMem(!memory), elems((T*)memory)
        {
            if (releaseMem)
            {
                memory = malloc(s * sizeof(T));
                elems = (T*)memory;
            }
            for (size_t i = 0; i < s; i++)
                T *temp = new(&elems[i]) T;
        }
        array(std::size_t s, const T &copyVal, void *memory = NULL) : sz(s), releaseMem(!memory), elems((T*)memory)
        {
            if (releaseMem)
            {
                memory = malloc(s * sizeof(T));
                elems = (T*)memory;
            }
            for (size_t i = 0; i < s; i++)
                T *temp = new(&elems[i]) T(copyVal);
        }
        ~array()
        {
            for (size_t i = 0; i < sz; i++)
                elems[i].~T();
            if (releaseMem)
                free(elems);
        }

        // size is constant
        size_type size() const { return sz; }
        bool empty() const { return sz != 0; }
        size_type max_size() const { return sz; }

        // swap (note: linear complexity)
        void swap (array<T,0>& y)
        {
            // error check
            std::swap_ranges(begin(),end(),y.begin());
        }

        // assignment with type conversion
        template <typename T2>
        array<T,0>& operator= (const array<T2,0>& rhs)
        {
            BOOST_ASSERT(rhs.size() == this->size());
            std::copy(rhs.begin(),rhs.end(), begin());
            return *this;
        }

};

    // comparisons
    template<typename T, std::size_t N>
    bool operator== (const array<T,N>& x, const array<T,N>& y) {
        return std::equal(x.begin(), x.end(), y.begin());
    }
    template<typename T, std::size_t N>
    bool operator< (const array<T,N>& x, const array<T,N>& y) {
        return std::lexicographical_compare(x.begin(),x.end(),y.begin(),y.end());
    }
    template<typename T, std::size_t N>
    bool operator!= (const array<T,N>& x, const array<T,N>& y) {
        return !(x==y);
    }
    template<typename T, std::size_t N>
    bool operator> (const array<T,N>& x, const array<T,N>& y) {
        return y<x;
    }
    template<typename T, std::size_t N>
    bool operator<= (const array<T,N>& x, const array<T,N>& y) {
        return !(y<x);
    }
    template<typename T, std::size_t N>
    bool operator>= (const array<T,N>& x, const array<T,N>& y) {
        return !(x<y);
    }

    // global swap()
    template<typename T, std::size_t N>
    inline void swap (array<T,N>& x, array<T,N>& y) {
        x.swap(y);
    }

} /* namespace kbf */

This is where I am currently.
This seems incredibly useful, and still very performant.
What I'd really like is to be able to eliminate vectors from most of my code, and use these things unless I really do need growable vectors...

I'm working on writing something up about how to use chained unwind info next, hopefully that won't be quite so long in coming as this post...

-Kev

How to get CL.exe proxy filters (like STLFilt) working under VS2005

Kevin Frei — Wed, 05 Apr 2006 18:33:00 GMT

There's a class of tools that lots of people write that replace cl.exe with a 'wrapper' to do things link pre-process a command line [perhaps adding particular -D arguments], or filtering the output to deal with things like psychotically complicated error messages from template metaprogramming. These tools tend to be incredibly useful, and also tend to be incredibly straightforward to author. And with VS2005, they tend to be broken when running under the IDE. With 2005, the compiler toolset finally got serious about supporting Unicode. You can write code in Unicode (so you can have variable names like "a picture of a telephone" :-) ) You should also get your error messages in Unicode. The trouble arises when you realize the ethnocentric history of software. It's the stomping ground of the English-speaking white male. I'm not trying to be rude, racist, sexist, or generally offensive, I'm just speaking historically (and unfortunately, it continues to be a field dominated by men, though the few women I have been able to work with are truly brilliant). Whoops - get off that tangent and back to the point... The Win32 console file handles cannot handle Unicode. They're code-page based handles, so when you're running under the console, we just spit out code-page based errors & warnings. If those errors & warnings contain Unicode, you're just out of luck. Well, when cl.exe is running under the IDE, we figured we could improve the experience. Instead of spitting messages out stdout and stderr, we open up some named pipes, and spit out Unicode that the IDE can then display. Which is great, until you start thinking about the cl.exe proxy scenario. So, if you're writing a cl.exe proxy, prior to invoking the original cl.exe, undefine VS_UNICODE_OUTPUT. This successfully tricks cl.exe into thinking you're just compiling from the command line, and you're back in business (with non-Unicode messages, though :-( )

Update: Leor Zolman (Mr. STLFilt himself!) mentioned he couldn't get this working. Turns out there's one more little issue that you may not realize: The response files being fed to cl.exe are probably unicode. You have to uncheck that, or else handle unicode response files for everything to work properly. If you turn off 'Use Unicode Response Files" in your project, go grab the source to STLFilt and change the doWin32Stuff function to have this at the beginning:

// from cl.cpp:
int doWin32Stuff(char *full_filt_cmd, char *cmdline)
{
    using namespace std;
    if (getenv("VS_UNICODE_OUTPUT"))
    {
        log("VS_UNICODE_OUTPUT is enabled - disabling!");
        // Clear the Unicode output flag to force the compiler to use the normal output pipes
        // I don't know perl at all, but this is how you do it from C:
        _putenv("VS_UNICODE_OUTPUT=");
    }

-Kev

x64 ABI vs. x86 ABI (aka Calling Conventions for AMD64 & EM64T)

Kevin Frei — Tue, 07 Mar 2006 02:00:00 GMT

(This is an older post, with some mild cleanup, and fixed links)

Before I start: ABI = Application Binary Interface – this is the spec that describes how to call functions, pass parameters, unwind the stack, handle exceptions, etc... It’s also sometimes call the ‘Calling Convention’

There is a persistent misconception among people who are implementing x64 compilers/code generators, and folks that write ASM code for x64, who have a functioning solution for x86. People too frequently assume they can just keep most of their code, while changing ESP to RSP, and things should 'just work'. This is fundamentally not true. When initially working on the x64 ABI, it was decided that we wanted to clean up the way that exception handling & general function invocation worked. We had a brand new architecture, we wanted to cut out all of the legacy junk that continually prevents, or at least overcomplicates, achieving great performance on the x86 platform. With this in mind, x64 was given a single calling convention – no __cdecl / __stdcall / __fastcall / __thiscall mess. There was also a dramatic change in the way that x64 unwinds the stack, compared to how x86 does it.

Unwinding the stack is used in a wide variety of places, including handling exceptions, garbage collection, and displaying the call stack from a debugger. On x86, every function that needs some sort of attention due to an exception must add an element to a thread-global linked list upon entry, and remove it upon exit. For non-exception unwinds, the thing doing the unwind must grok through some nasty meta-data that tries to describe what & when the compiler is setting up the stack frame. This meta-data was only implemented after about 1999, and is primarily supported in debuggers. I’ll be ignoring this junk, and instead focus on how exception handling works, since this is the primary way your code will break if you don’t get this right (undebuggable code is still broken, but you won’t notice until you try to get a stack trace).

So, the x86 thread-global linked list contains a list of structures, each element of such contains a function pointer to call in the event of an exception, and then some data that said function will consume. Thus, you’ll see fs:[0] references scattered throughout C++ code:every function that contains a destructor that must be invoked if an exception occurs must have one of these things. When your code creates a new object, there is a small bit of code executed to update the data on this thread-global list element. When that object is destroyed, more code is executed. Because of this, x86 can catch hundreds of thousands of exceptions per second, but if you don’t actually take any exceptions, your CPU executes lots of blobs of code that have no purpose except to be sure that the rare exception is handled properly. Finding the first function that needs to handle an exception, or destroy an object is an O(1) operation: it’s a single lookup of FS:[0]. VERY fast. Unfortunately, that should not be a common scenario. Exceptions should be exceptional, not common!In addition, this linked list really resides on the stack, thus there is a function pointer sitting right below the return address on your stack!Buffer overruns, anyone?There is now an extra blob of information that indicates what functions are valid to be invoked as exception handlers (link /SAFESEH, if you’re curious), but this is only used if every contribution to your .exe or .dll has this information.

With this in mind, on x64 every function has a very strict structure that must be properly described in static data. The prolog is the only place in which you can adjust your stack frame pointer. The prolog can only be up to 255 bytes long. All modifications of your stack frame pointer, as well as all saves of nonvolatile registers (RBP, RSI, RDI, R12-R15, XMM6-XMM15) must be described in this static data, so that they can be restored correctly if an exception occurs. If you function is missing this static data, and an exception is raised, the thread will be terminated by the OS.

In an effort to get stuff into peoples hands, here’s an excerpt that I’ve prepended to the ABI document that has not yet seen the light of day. I’ve updated the links to point you

at the sections on MSDN. Rather than posting the whole thing, I've just put in my description, with links back to the slightly older version on MSDN. The current version has some more detail, but nothing you couldn't figure out through, ahem, trial & error...

<snip>

Overview

The in depth nature of an ABI document doesn’t lend itself to ‘easy reading’. However, it is the case that a detailed knowledge of the entire ABI is rarely necessary to accomplish most programming tasks. This section is simply a quick overview of the ABI, with pointers to the sections that describe the various aspects in more detail. It also tries to point out particular ‘gotchas’ that must be strictly adhered to, in an effort to minimize the problems encountered.

Calling convention

The x64 Windows ABI is a 4 register ‘fast-call’ calling convention, with stack-backing for those registers. There is a strict one-to-one correspondence between arguments in a function, and the registers for those arguments. Any argument that doesn’t fit in 8 bytes, or is not 1, 2, 4, or 8 bytes, must be passed by reference. There is no attempt to spread a single argument across multiple registers. The x87 register stack is unused. It may be used, but must be considered volatile across function calls. All floating point operations are done using the 16 XMM registers. The arguments are passed in registers RCX, RDX, R8, and R9. If the arguments are float/double, they are passed in XMM0L, XMM1L, XMM2L, and XMM3L. 16 byte arguments are passed by reference. Parameter passing is described in detail at MSDN Library: Dev Tools and Languages: VS2008: VS: VC++: 64-bit Programming: x64 Software Conventions: Calling Convention: Parameter Passing. In addition to these registers, RAX, R10, R11, XMM4, and XMM5 are volatile. All other registers are non-volatile. Register usage is documented in detail at MSDN Library: Dev Tools and Languages: VS2008: VS: VC++: 64-bit Programming: x64 Software Conventions: Register Usage and MSDN Library: Dev Tools and Languages: VS2008: VS: VC++: 64-bit Programming: x64 Software Conventions: Calling Convention: Caller/Callee Saved Registers.

The caller is responsible for allocating space for parameters to the callee, and must always allocate sufficient space for the 4 register parameters, even if the callee doesn’t have that many parameters. This aids in the simplicity of supporting K&R C’s unprototyped functions, and ‘vararg’ C/C++ functions. For vararg/unprototyped functions any float values must be duplicated in the corresponding general-purpose register. Any parameters above the first 4 must be stored on the stack, above the backing-store for the first 4, prior to the call. Vararg function details can be found at MSDN Library: Dev Tools and Languages: VS2008: VS: VC++: 64-bit Programming: x64 Software Conventions: Calling Convention: Varargs. Unprototyped function information is detailed at MSDN Library: Dev Tools and Languages: VS2008: VS: VC++: 64-bit Programming: x64 Software Conventions: Calling Convention: Unprototypesd Functions.

Alignment

Most structures are aligned to their natural alignment. The primary exceptions are the stack pointer and malloc/alloca memory, which are aligned to 16 byte, in order to aid performance. Alignment above 16 bytes must be done manually, but since 16 bytes is a common alignment size for XMM operations, this should suffice for most code. For more information about structure layout and alignment see MSDN Library: Dev Tools and Languages: VS2008: VS: VC++: 64-bit Programming: x64 Software Conventions: Types and Storage. For information about the stack layout, see MSDN Library: Dev Tools and Languages: VS2008: VS: VC++: 64-bit Programming: x64 Software Conventions: Stack Usage.

Unwindability

All non-leaf functions [functions that neither call a function, nor allocate any stack space themselves] must be annotated with data [referred to as xdata or ehdata, which is pointed to from pdata] that describes to the operating system how to properly unwind them, to recover non-volatile registers. Prologs & epilogs are highly restricted, so that they can be properly described in xdata. The stack pointer must be aligned to 16 bytes, except for leaf functions, in any region of code that isn’t part of an epilog or prolog. For details about the proper structure of function prolog & epilogs, see MSDN Library: Dev Tools and Languages: VS2008: VS: VC++: 64-bit Programming: x64 Software Conventions: Prolog and Epilog. For more information about exception handling, and the exception handling/unwinding pdata & xdata see MSDN Library: Dev Tools and Languages: VS2008: VS: VC++: 64-bit Programming: x64 Software Conventions: Exception Handling (x64).

</snip>

Anyway, I hope that helps. I could not believe how long it took to find this stuff on MSDN. For some (lame) reason, blogs are better indexed than MSDN in both MSN search, and Google. Hopefully this entry will make finding this information easier.

Here’s a link to the official x64 ABI documentation, which goes into excruciating detail about this stuff. Sorry if it’s not very readable – I wrote a few parts, along with a few other people, and we’re primarily engineers, not writers. We have had a great UE write take over on this document, and it should be slightly improved when it sees the light of day as part of the VS 2005 documentation, but until then, this is it:

MSDN Library: Dev Tools and Languages: VS2008: VS: VC++: 64-bit Programming: x64 Software Conventions

Hopefully, that link will continue to work for a while. MSDN likes to occasionally restructure the way it references information, just to keep us all on our toes. I've now updated the links 3 times. If they're broken, please ping me and I'll update them!

Why isn't __assume on for asserts in retail mode?

Kevin Frei — Tue, 31 Jan 2006 01:48:00 GMT

__assume and __forceinline were the first two features I implemented as a developer out of college. I just resolved a bug as 'By Design' that asked the question posed in the title. Here's a brief explanation:

First, __assume(n==x) basically tells the expression optimizer & the global optimizer in the compiler that n is being assigned the value of x. So, if you say __assume(n==5);y=n*n; the compiler will automagically turn that into y=25. Nifty, right? It's also used to flag dead code:

switch(i) {
case 1:
case 2:
    printf("Hello\n");
    break;
default:
    __assume(0);
}

For that switch, there will be no checks for any values other than 1 or 2 - no range check. Again, in performance sensitive code, this can be a big win.

However, the transformation of assert to __assume could have potentially negative impact on security, so it's not defined by default. Individual customers should determine if it's an acceptable optimization.

The trouble lies in the potential lack of code coverage testing. Here's a simplistic case to illustrate the problem:

void untestedFunctionFoo()
{
    assert(passwordAlreadyChecked == true);
    // Assert is actually false,
    // but we don't realize that,
    // because we haven't tested it
    DoSomethingThatChecksSecurity();
}

void DoSomethingThatChecksSecurity()
{
    if (!passwordAlreadyChecked)
        GetPassword();
    doSomethingSecure();
}

If the compiler inlines DoSomethingThatChecksSecurity into untestedFunctionFoo, and assert is turned into __assume(passwordAlreadyChecked == true) the optimizer will remove the call to GetPassword(), and then untestedFunctionFoo, which wasn't tested, no longer requires a password to get it. Your visual code reviews don't pick up that doSomethingSecure might occur without a password, because logically, you're assertting that the password has already been checked, or you're calling GetPassword.

Hopefully, that explains the issue. Please use __assume with caution. It can be a performance win, but if you're authoring code with severe security concerns, you're better off without it.

-Kev

X64 Unwind Information

Kevin Frei — Thu, 05 Jan 2006 00:48:00 GMT

I've had a fairly large number of e-mails with various people both inside & outside Microsoft explaining the AMD64 unwind data. I generally push them at the ABI documentation (which I've linked to in this entry). But the ABI documentation really requires a complete reading before you can really understand how the unwind information all fits together. So, in an attempt to make dealing with unwind information, I'm going to attempt a more 'chatty' explanation of unwind data.

Background

The 2 phase exception model

Both Win32 & Win64 have a 2 phase exception model. The first phase walks the list of functions that _might_ handle an exception, asking them if they will _actually_ handle the particular exception that occurred (this is done by calling filter functions). During this phase, the filter functions either return EXCEPTION_CONTINUE_SEARCH (0) which indicates that this handler will not handle the exception, EXCEPTION_HANDLE_EXCEPTION (1), which indicates that this handler will handle the exception. One other option is EXCEPTION_RESUME_EXECUTION (-1) that I won't mention (look it up in MSDN if you're curious).

Why do you need unwind data?

First, you should probably understand WHY the Windows AMD64 ABI requires unwind data. In Win32 x86 land, when an exception occurs, the OS just looks at a pointer at fs:0 which is the head of a linked list of information regarding what to do in the event of an exception. Each function that requires any sort of attention when an exception occurs needs to create a node on this linked list. This node must contain all information necessary to handle the two phase exception mechanism of Win32. For Windows for AMD64, the way the functions that need to be invoked during exception handling are discovered not by walking a linked list, but crawling the stack. Thus, the stack must _always_ be in a state that can be statically walked. To accomplish this, there are 2 fundamental issues: how to discover the stack frame size of a function, and how to recover non-volatile (aka caller saved) register values. This is exactly what AMD64 unwind data describes. The trouble most people run into, though, is that even if your function doesn't need to do anything if an exception occurs, the function may be called by a function that does, and it may then call a function that will throw an exception. If this is the case, when the exception is thrown, the function's stack frame must be fully described. As an added bonus, because all stack frames are accurately described, there's never any reason to use a frame pointer unless absolutely necessary due to something like _alloca or __declspec(align(>16)).

So, even if you just have a tiny little function that only calls another function, you still need unwind data, or when an exception occurs, your process will simply be terminated.

OK, What is unwind data?

Simply put, it's a meta-language that describes what, when, and how a function's frame is built. There are opcodes that indicate when & by how much the stack pointer has been adjusted, when & where a non-volatile register has been saved, or when, where, and to what offset a frame pointer has been set. When you're writing assembly code for ML64, there are predefined macros to help describe this stuff:

PROC FRAME [optional handler address]

.ALLOCSTACK size

.PUSHFRAME [code]

.PUSHREG reg

.SAVEREG reg, offset

.SAVEXMM128 reg, offset

.SETFRAME reg, offset

.ENDPROLOG

Note that all those offsets are actually restricted to be properly aligned. The .SAVEREG offset must be a multiple of 8, while .SAVEXMM128 offset must be a multiple of 16. The .SETFRAME offset must be a multiple of 16, and must be between 16 and 240. In addition, all frame manipulation must be completed in the first 254 bytes of the function. If you want to push registers saves & restores further into the function, you must use chained unwind info, which will have to be the subject of another blog entry...

Looking up the directives on MSDN will give you examples of how they're all used. If you feel like authoring code that conforms to the prologue unwind descriptors is restrictive, you'll love the epilogue requirements. All function epilogues must look like this:

(optional) lea rsp, [frame ptr + frame size] or add rsp, frame size

pop reg (zero or more)

ret (or jmp)

No other instructions may occur betwen the first lea/add and the final jmp or ret. At first glance, this may seem like you can't restore any XMM registers. The trick is that all non-volatile registers except those that you want to restore using a pop must be restored prior to entry to the epilogue. The reason this works is that if the OS has to unwind an epilogue, it already has the correct values in all the registers except the ones that are restored via pop, so things really do work out well.

One other note: if the final jmp isn't an ip-relative jmp, but an indirect jmp, it must be preceded by the REX prefix, to indicate to the OS unwind routines that the jump is headed outside of the function, otherwise, the OS assumes it's a jump to a different location inside the same function.

From here, I think I might see what kind of questions pop up, and go from there. I'll also add a future entry to describe how to use chained unwind information to allow you to save registers later in the function that the first 254 bytes (although I _think_ this requires that you need to author your own .pdata & .xdata which is a whole lot more complicated...)

-Kev

How to make a no-inline inline function (and why you might want to)

Kevin Frei — Wed, 26 Oct 2005 21:20:00 GMT

The Windows folks had a problem with the AMD64 compiler last week. The interference package is not quite as good as the x86 one, so when it sees something like this:

// header.h

inline void foo(int a, int b, int c)

{

bar(&a, &b, &c);

}

// function.cpp

#include "header.h"

void func()

{

foo(1,2,3);

foo(4,5,6);

}

we inline foo into func twice, generate (because they're address is taken) temporary storage for 1,2,3 & 4,5,6, but then we don't realize that, because they're scope-local address taken variables (so any global function that may be using these addresses after the end of the foo scope is broken), we should be able to pack the two sets of temp's together. This resulted in a kernel stack overflow (very very very bad).

The solution is to prevent the compiler from ever inlining foo, so that the temps are never allocated in func's stack frame. The trouble with this, is that if you remove the inline, and add a __declspec(noinline) to the foo implementation, each .C file that includes header.h gets a new copy of foo, and when you try to link, you get a multiply defined symbol error. For data, the proper way to handle this is with __declspec(selectany), which tells the linker that if you see multiple symbols all claiming to define the same name, just pick any one of them. Except you can't use selectany on a function definition, only on a variable definition.

So, instead, you use __declspec(noinline) inline void foo(...). While it's not intuitive (and kind of funny) what's going on, in order for inline functions to be defined in a header, they require the selectany attribute, so the compiler, when it sees the 'inline' C++ keyword, tags the function as a 'selectany' section for the linker. Problem solved!

The primary reasons I can see to do this, (besides the reason the Windows Kernel folks have), is if you want to have some mechanism in place that is completely handled in headers, but you don't want to bloat your code or you want to be able to see, from disassembly, where the function is being called. Can someone come up with other reasons?

Boost & BJam plus Web Chats coming up

Kevin Frei — Tue, 09 Nov 2004 06:07:00 GMT

I'm a big C++ fan, despite the fact that the VC++ optimizer is written in C. I've been using Boost for a couple years now, and just picked up the initial tarball for 1.32.0. Someone's written a .jam file for VC8 on x64, but it just does the bare minimum - lots of warnings & whatnot. I have written these for all 3 platforms and I've been tempted to try my hand at a PocketPC ARM & x86 version, too (since I fool around on those platforms, occasionally). I haven't ever bothered testing the IA64 stuff, but I haven't had any problems with my x64 & x86 version. I'll have to figure out if I can upload files here. Anyway, for now I'll just toss in the x64 one I've got inline:

# Microsoft Visual C++

# (C) Copyright David Abrahams 2001. Permission to copy, use,
# modify, sell and distribute this software is granted provided this
# copyright notice appears in all copies. This software is provided
# "as is" without express or implied warranty, and with no claim as
# to its suitability for any purpose.

# The following #// line will be used by the regression test table generation
# program as the column heading for HTML tables. Must not include version number.
#//<a href="http://msdn.microsoft.com/vstudio/default.asp">Microsoft VC++</a>

# singleton variables...
set-as-singleton VC8_ROOT VC_SETUP ;

# Get these variable set on the targets so that we can re-use the
# build actions for other toolsets using this one as a base.
flags vc80x64 VC_SETUP ;
flags vc80x64 VC_COMPILER ;
flags vc80x64 VC_LINKER ;
flags vc80x64 VC_PDB_NAME ;

# compute vc80x64 tool path

VC8_ROOT = $(VS80COMNTOOLS:J=" ")"..\\..\\VC" ;
VC8_ROOT = $(VC8_ROOT:J=" ") ;

VC_SETUP = "CALL \"$(VC8_ROOT)$(SLASH)vcvarsall.bat\" x86_amd64 >nul" ;

VC_COMPILER = cl ;
VC_LINKER = link ;
VC_PDB_NAME = vc80x64 ;
VC_STDLIB_PATH = ;

flags vc80x64 CFLAGS <debug-symbols>on/<debug-store>object : /Z7 ;
flags vc80x64 CFLAGS <debug-symbols>on/<debug-store>database : /Zi ;
flags vc80x64 PDB_CFLAG <debug-symbols>on/<debug-store>database : /Fd ;
flags vc80x64 PDB_LINKFLAG <debug-symbols>on/<debug-store>database : /PDB: ;
flags vc80x64 LINKFLAGS <debug-symbols>on : /DEBUG ;

# The linker disables the default optimizations when using /DEBUG. Whe have
# to enable them manually for release builds with debug symbols.
flags vc80x64 LINKFLAGS <debug-symbols>on/<runtime-build>release : /OPT:REF,ICF ;

flags vc80x64 CFLAGS <optimization>off : /Od ;
flags vc80x64 CFLAGS <optimization>speed : /O2 ;
flags vc80x64 CFLAGS <optimization>space : /O1 ;
flags vc80x64 CFLAGS <inlining>off : /Ob0 ;
flags vc80x64 CFLAGS <inlining>on : /Ob1 ;
flags vc80x64 CFLAGS <inlining>full : /Ob2 ;
flags vc80x64 CFLAGS <exception-handling>on : /EHsc ;
flags vc80x64 CFLAGS <rtti>on : /GR ;

# Note that these two options actually imply multithreading support on vc80x64
# because there is no single-threaded dynamic runtime library. Specifying
# <threading>multi would be a bad idea, though, because no option would be
# matched when the build uses the default settings of <runtime-link>dynamic
# and <threading>single.

flags vc80x64 CFLAGS <runtime-build>release/<runtime-link>dynamic : /MD /GL ;
flags vc80x64 LINKFLAGS <runtime-build>release/<runtime-link>dynamic : /LTCG ;
flags vc80x64 CFLAGS <runtime-build>debug/<runtime-link>dynamic : /MDd ;

# VC8 doesn't support a single threaded runtime, just some single-threaded optimizations
flags vc80x64 CFLAGS <runtime-build>release/<runtime-link>static/<threading>single : /MT /D_ST_MODEL ;
flags vc80x64 CFLAGS <runtime-build>debug/<runtime-link>static/<threading>single : /MTd /D_ST_MODEL ;

flags vc80x64 CFLAGS <runtime-build>release/<runtime-link>static/<threading>multi : /MT ;
flags vc80x64 CFLAGS <runtime-build>debug/<runtime-link>static/<threading>multi : /MTd ;

flags vc80x64 CFLAGS <cflags> ;
flags vc80x64 C++FLAGS <cxxflags> ;
flags vc80x64 DEFINES <define> ;
flags vc80x64 UNDEFS <undef> ;
flags vc80x64 HDRS <include> ;
flags vc80x64 STDHDRS <sysinclude> ;
flags vc80x64 LINKFLAGS <linkflags> ;
flags vc80x64 ARFLAGS <arflags> ;

flags vc80x64 STDHDRS : $(VC8_ROOT)$(SLASH)include ;
flags vc80x64 STDLIBPATH : $(VC8_ROOT)$(SLASH)lib$(SLASH)amd64 ;
flags vc80x64 LIBPATH <library-path> ;
flags vc80x64 NEEDLIBS <library-file> ;
flags vc80x64 FINDLIBS <find-library> ;
flags vc80x64 LINKFLAGS <target-type>$(SHARED_TYPES) : /DLL ;

flags vc80x64 LINKFLAGS <user-interface>console : /subsystem:console ;
flags vc80x64 LINKFLAGS <user-interface>gui : /subsystem:windows ;
flags vc80x64 LINKFLAGS <user-interface>wince : /subsystem:windowsce ;
flags vc80x64 LINKFLAGS <user-interface>native : /subsystem:native ;
flags vc80x64 LINKFLAGS <user-interface>auto : /subsystem:posix ;

rule vc-set-pdb-file ( targets + : name )
{
    local pdb = $(targets[1]:B=$(name):S=.pdb) ;
    VC_PDB_FILE on $(targets) = $(pdb:G=:R=$(LOCATE_TARGET)) ;
    LOCATE on $(pdb) = $(LOCATE_TARGET) ;
    Clean clean : $(pdb) ;
}

#### Link ####

rule Link-action ( target implib ? : sources + : target-type ? )
{
with-command-file vc-Link $(<) : $(sources) $(NEEDLIBS) ;

    gRUN_PATH($(<)) += $(VC_STDLIB_PATH) ;
    if $(implib)
    {
        # incremental linking a DLL causes no end of problems: if the
        # actual exports don't change, the import .lib file is never
        # updated. Therefore, the .lib is always out-of-date and gets
        # rebuilt every time. I'm not sure that incremental linking is
        # such a great idea in general, but in this case I'm sure we
        # don't want it.
        NOINCREMENTAL on $(<) = /INCREMENTAL:NO ;
    }
    vc-set-pdb-file $(<) : $(target:B) ;
}

actions together vc-Link
{
$(VC_SETUP)
$(VC_LINKER) /nologo $(NOINCREMENTAL) $(LINKFLAGS) $(PDB_LINKFLAG)"$(VC_PDB_FILE)" /out:"$(<[1])" /IMPLIB:"$(<[2])" /LIBPATH:"$(LIBPATH)" /LIBPATH:"$(STDLIBPATH)" "$(FINDLIBS:S=.lib)" @"$(>)"
}

#### Cc #####

rule Cc-action
{
vc-set-pdb-file $(<) : $(VC_PDB_NAME) ;
vc-Cc $(<) : $(>) ;
}

actions vc-Cc
{
$(VC_SETUP)
$(VC_COMPILER) /Zm800 -nologo -wd4996 -wd4103 -c -U$(UNDEFS) -D$(DEFINES) $(CFLAGS) -I"$(HDRS)" -I"$(STDHDRS)" $(PDB_CFLAG)"$(VC_PDB_FILE)" -Fo"$(<)" "$(>)"
}

#### C++ ####
rule C++-action
{
vc-set-pdb-file $(<) : $(VC_PDB_NAME) ;
vc-C++ $(<) : $(>) ;
}

actions vc-C++
{
$(VC_SETUP)
$(VC_COMPILER) /Zm800 -nologo -wd4996 -wd4103 -EHsc -c -U$(UNDEFS) -D$(DEFINES) $(CFLAGS) $(C++FLAGS) -I"$(HDRS)" -I"$(STDHDRS)" $(PDB_CFLAG)"$(VC_PDB_FILE)" -Fo"$(<)" -Tp"$(>)"
}

#### Archive ####
rule Archive-action
{
vc-set-pdb-file $(<) : $(<:B) ;
with-command-file vc-Archive $(<) : $(>) ;
}

actions vc-Archive
{
    $(VC_SETUP)
    if exist "$(<)" DEL "$(<)"
    $(VC_LINKER) /lib $(ARFLAGS) $(PDB_LINKFLAG)"$(VC_PDB_FILE)" /out:"$(<)" @"$(>)"
}

That seems to work pretty well. I haven't tried this with Beta 1 - I did the work @ home, but didn't test it until I got to work, so I only ran against a more recent build.

Microsoft will be hosting an online chat November 9, 2004 at 9:00am Pacific Time to answer your Managed Code on 64-bit Windows Platforms questions.

http://msdn.microsoft.com/chats/

http://msdn.microsoft.com/chats/outlook_reminders/04_MC_64BitWP_Nov9.ics

And I'll be one of the hosts in this one on Wednesday, November 10th at 9AM PST:

http://msdn.microsoft.com/chats/outlook_reminders/04_MC_64Bit_Nov10.ics

I also answered questions in a couple of excellent 'WebCasts' hosted by Kang Su Gatlin, one of our project managers on the compiler, last week that people might be interested in. If I ever find the stinking links to them, I'll put them up here, too.

First Post - all things x64

Kevin Frei — Fri, 05 Nov 2004 23:24:00 GMT

Welcome to my stinkin' blog. I just wanted a place to dump interesting & useful blobs of information about what I've been doing for the past 4 or 5 years of my life @ work: x64.

My name is Kevin Frei, and I've been a developer on the x64 (aka AMD64, EM64T, and x86-64) compiler since it's inception. I'm originally from Pocatello, ID, I really don't miss it one bit. Except maybe the cost of housing :-)

So, here's the stuff I wanted to make public. It's all generally about how to successfully use Windows XP Pro for x64 (and Win2003 Server for x64) in a normal day-to-day environment. I'll probably toss out useful compiler/code gen things in the future, but for right now, this is it:

So, without further ado, here we go:

%windir%\syswow64\cmd.exe is your best friend

When you launch the 32 bit command prompt, PROCESSOR_ARCHITECTURE is set to x86, ProgramFiles is set to "c:\program files (x86)", and suddenly lots of things 'just work'

Use Internet Explorer (32 bit)

The 64 bit internet explorer works great, unless you need to run activeX controls, which are present on far too many web pages out there. Of course, the 64 bit version of IE is kind of nice, simply because there's no more irritating Flash advertisements, and no obnoxious addins. I've actually started using Avant Browser, and have found that it's quite pleasant, though always 32 bit...

'Program Files (x86)' can cause lots of problems with scripts

If you have scripts that use %ProgramFiles%, this can act a little weird, due to parentheses parsing problems. You can work your way around it by doing this from inside your script:

for %%D in ("%ProgramFiles%") do set ProgramFiles=%%~sD

This makes the variable use short pathnames: no more parenthese or spaces. It's actually a useful trick in general to get spaces & parentheses out of environment variables, in general. Maybe I'll fix mine fixer script and put it up here in the future...

Detecting that you're running under WOW64 from 32 bit scripts

PROCESSOR_ARCHITEW6432 is set to the real processor architecture when you're running syswow64\cmd.exe

When is system32 not system32?

When it's been mapped to syswow64! If you're running under WoW64, your actual system32 directory isn't visible - it's a duplicate of syswow64. Magic! Sometimes this can be little confusing, though.

How can I see stuff in the real system32 from a 32 bit process?

From a 64 bit CMD, linkd [somewhere] %windir%\system32

Now, from 32 bit apps, you can see the contents of the real system32 [with all it’s 64 bit images] from a 32 bit process, from the [somewhere] directory. I don't know if linkd.exe is available outside of MSFT. I'm sure there's some other way to accomplish this...