Slava Oks's WebLog : Coding & Debugging tips

Retrieving information about CPUs' relationships

slavao — Mon, 13 Mar 2006 00:52:00 GMT

In order for server side applications to run efficiently on modern hardware they have to understand hardware layout. As we are moving towards multicore and multi node NUMA systems, understanding relationships amongst cores, logical and physical CPU has...(read more)

Be aware: To Hyper or not to Hyper

slavao — Sat, 12 Nov 2005 20:48:00 GMT

Our customers observed very interesting behavior on high end Hyperthreading (HT) enabled hardware. They noticed that in some cases when high load is applied SQL Server CPU usage increases significantly but SQL Server performance degrades. Occasionally they would also see message in errorlog indicating one of the threads can't acquire a spinlock (Spinlock is a lightweight synchronization data structure. Threads keep a spinlock for short period of time. If a thread can't acquire a spinlock it will spin in a tight loop waiting for spinlock to become available). Customers also noticed that when HT is disabled under the same circumstances CPU usage increases slightly and performance is at appropriate level. What is going on here?

After some pondering and investigation we came up with the following theory. Database server has different type of threads: worker threads, threads executing clients request, and system threads that performing system tasks. An example of a system thread is a lazywriter. The responsibility of the lazywriter is to lazily scan through database buffers and push least recently used buffers out of the cache. Lazywriter usually kicks in whenever data page cache is full and server short of free data page buffers. The interesting behavior of lazywriter is that it scans large amount of memory. With Intel HT technogolgy logical processors share L1 & L2 caches. As you would guess lazywriter's behavior can potentially trash L1 & L2 caches. If a worker thread end up running on logical CPU that shares physical CPU with lazywriter its cache will be constantly trashed. It means that most of the memory accesses for the worker thread will be L1 & L2 misses. Moreover whenever the worker attempts to access hot data structure protected by spinlock it potentially might have a less chance of acquiring the spinlock which will cause the worker's spinning. This behavior will translate to high CPU utilization and significant drop in performance.

To confirm the theory I decided to create an experiment. I wrote a test program which consist of two types of threads: scan threads and lock threads. Scan threads are scanning through large amounts of memory. Lock threads constantly attempt to acquire global lock. If the theory is correct a lock thread that shares a physical CPU with a scan thread should acquire lock much less often than other threads in a given period of time. Below is the program's listing. (Disclosure: The program is for illustrative purpose only, it was written to illustrate the observed behavior, it shouldn't be treated as a final-well tested product :-))

#include <stdio.h>

#include <stdlib.h>

#include <windows.h>

#include <eh.h>

#include <wtypes.h>

#include <locale.h>

#include <process.h>

extern "C" void _mm_pause(void);

#pragma intrinsic (_mm_pause)

inline void

SmtPause ()

{

#if defined (_X86_) || defined (_AMD64_)

// PAUSE instruction (REP NOP) helps SMT scheduling

_mm_pause ();

#endif // _X86_

}

// Statistic information for threads

struct Statistics

{

int locked;

int spinned;

DWORD_PTR affinity;

};

// Type of threads the tests using

enum ThreadTypes

{

SCAN_THREAD,

LOCK_THREAD,

LAST_THREAD

};

// Entry Point

typedef void( __cdecl *ThreadEntryPoint )( void * );

// Inforamtion per thread - enables us to create threads in a common way

struct ThreadInfo

{

ThreadEntryPoint func;

LONG_PTR affinity;

};

// Global constants

// Number of CPUs

const int MaxNumberOfCPUs = 64;

// Block of memory to scan by scan thread

char MemoryBlock [1024 *1024];

// Stat information: number of threads is limmited by number of CPU,

// that is the purpose of the test

volatile Statistics Stats [MaxNumberOfCPUs];

// Number of threads running

volatile long NumberOfThreadsRunning = 0;

// "Spinlock" that all lock threads attempt to acquire

volatile long Lock = 0;

//-----------------------------------------------------------------------------

// Function: scanThread

// Description:

// Attempt to invalidate caches

// Notes:

// 1. Function changes 1MB of contiguous vm memory which appear

// to be enought invalidate L1&L2 caches

void __cdecl

scanThread (void* param)

{

char i = 0;

ULONG_PTR threadAffinity =(DWORD_PTR) param;

long threadId = InterlockedIncrement (&NumberOfThreadsRunning) -1;

SetThreadAffinityMask (GetCurrentThread (),threadAffinity);

// Initialize affinity

Stats [threadId].affinity = threadAffinity;

while (i >= 0)

{

// Memset memory: quick and dirty scan

memset (MemoryBlock, 0, sizeof (MemoryBlock));

Stats [threadId].spinned++;

}

return;

}

//-----------------------------------------------------------------------------

// Function: lockThread

// Description:

// Acquires and releases a lock in a loop

// Notes:

void __cdecl

lockThread (void* param)

{

ULONG_PTR threadAffinity =(DWORD_PTR) param;

long threadId = InterlockedIncrement (&NumberOfThreadsRunning)-1;

SetThreadAffinityMask (GetCurrentThread (), threadAffinity);

// Initialize affinity

Stats [threadId].affinity = threadAffinity;

while (1)

{

if (Lock == 0 && InterlockedExchange (&Lock, 1) == 0)

{

// Increment number of times we acquired lock

Stats [threadId].locked ++;

Lock = 0;

}

else

{

Stats [threadId].spinned++;

// Be HT friendly

SmtPause ();

continue;

}

return;

}

//-----------------------------------------------------------------------------

// Function:main

// Description:

// Creates lock and scan threads, reporting stats every 10sec

// Notes:

int __cdecl

main (

int argc,

char* argv [])

{

LONG_PTR threadId = 0;

ThreadInfo threadInfo [LAST_THREAD];

DWORD_PTR processAffinity;

DWORD_PTR threadAffinity;

DWORD_PTR systemAffinity;

int counter = 0;

int numberOfThreads = 0;

// Retrive process affinity

if (GetProcessAffinityMask (GetCurrentProcess (),

&processAffinity,

&systemAffinity) == FALSE)

{

printf ("GetProcessAffinityMask failed - exiting\n");

exit (-1);

}

// Initialize threads Information: First default and then check

// if client passed anything in

// Allocate all but one CPUs for threads running locking code

threadInfo [LOCK_THREAD].func = lockThread;

threadInfo [LOCK_THREAD].affinity = processAffinity & processAffinity-1;

// Allocate one CPU for scan thread

threadInfo [SCAN_THREAD].func = scanThread;

threadInfo [SCAN_THREAD].affinity =threadInfo [LOCK_THREAD].affinity ^ processAffinity;

// Read command line arguments: in simple way

for (counter = 0 ; counter< argc; counter++)

{

switch (*(argv [counter] + 1))

{

case 's':

case 'S':

{

// Retrieve affinity mask for scan threads

threadInfo [SCAN_THREAD].affinity =

(DWORD_PTR) _atoi64 ((argv [counter] + 2));

break;

}

case 'l':

case 'L':

{

// Retrieve affinity mask for lock threads

threadInfo [LOCK_THREAD].affinity =

(DWORD_PTR) _atoi64 ((argv [counter] + 2));

break;

}

case '?':

{

printf ("Usage: test -sn -lm \n"

"Example: test.exe -s8 -l7 (runs 1 scanning thread and 3 locking threads)\n"

"-sn is affinity of threads performing memory scan \n"

"-sm is affinity of threads acquiring lock");

exit (0);

}

default:

{

break;

}

// Start threads of all types based on supplied affinities

for (counter = 0; counter < LAST_THREAD; counter++)

{

while (threadInfo [counter].affinity != 0)

{

// Separate affiinity for a thread we are trying to

// create from all affinity mask

// Remember full mask

threadAffinity = threadInfo [counter].affinity;

// Turn off least significant bit

threadInfo [counter].affinity &= threadInfo [counter].affinity -1;

// Calculate the affinity bit for the thread:

// Leave one bit on - affinity bit

threadAffinity ^= threadInfo [counter].affinity;

// Use beginthread for simplicity

if (_beginthread (threadInfo [counter].func,

NULL,

(void*) threadAffinity) == -1)

{

printf ("Failed to create thread - exiting\n");

exit (-1);

}

numberOfThreads++;

}

// Loop and report stats periodically

while (1)

{

// Report every 10 sec

Sleep (10000);

for (threadId=0; threadId < numberOfThreads; threadId++)

{

printf ("Thread Id = %d\t, Thread Affinity = %p\t, Locked = %d\t, Spin = %d\t\n ",

threadId,

Stats [threadId].affinity,

Stats [threadId].locked,

Stats [threadId].spinned);

// Reset run time data

Stats [threadId].locked = 0;

Stats [threadId].spinned =0;

}

return 0;

}

As you can see one can specify an affinity for scan and lock threads at start up of the program.

I ran sever experiments on my dev box which has 2 physical and 4 logical CPUs. CPU 1 (Affinity 0x1) & 3 (Affinity 0x4) and 2 (Affinity 0x2) & 4 (Affinity 0x8) share physical CPUs respectively. The box also has Windows 2003 Server SP1 on it. Below are the results:

test.exe -s8 -l7

Thread Id = 0 , Thread Affinity = 00000008 , Locked = 0 , Spin = 9035

Thread Id = 1 , Thread Affinity = 00000001 , Locked = 9290331 , Spin = 10373717

Thread Id = 2 , Thread Affinity = 00000002 , Locked = 676397 , Spin = 40879794

Thread Id = 3 , Thread Affinity = 00000004 , Locked = 10274030 , Spin = 42433521

test.exe -s4 -l11

Thread Id = 0 , Thread Affinity = 00000004 , Locked = 0 , Spin = 8943

Thread Id = 1 , Thread Affinity = 00000001 , Locked = 741166 , Spin = 11421139

Thread Id = 2 , Thread Affinity = 00000002 , Locked = 10747824 , Spin = 10024252

Thread Id = 3 , Thread Affinity = 00000008 , Locked = 10210221 , Spin = 13988134

test.exe -s1 -l14

Thread Id = 0 , Thread Affinity = 00000001 , Locked = 0 , Spin = 8472

Thread Id = 1 , Thread Affinity = 00000002 , Locked = 10550011 , Spin = 20156206

Thread Id = 2 , Thread Affinity = 00000004 , Locked = 722554 , Spin = 11211074

Thread Id = 3 , Thread Affinity = 00000008 , Locked = 11182506 , Spin = 25376166

test.exe -s2 -l13

Thread Id = 0 , Thread Affinity = 00000002 , Locked = 0 , Spin = 8585

Thread Id = 1 , Thread Affinity = 00000001 , Locked = 9900899 , Spin = 12267885

Thread Id = 2 , Thread Affinity = 00000004 , Locked = 8984095 , Spin = 11909080

Thread Id = 3 , Thread Affinity = 00000008 , Locked = 1297138 , Spin = 28199769

Great! The experiment confirms the theory. So does it mean you have to disable HT when using SQL Server? The answer is it really depends on the load and hardware you are using.

You have to test your application with HT on and off under heavy loads to understand HT's implications.

Keep in mind that not only lazywriter thread can cause slowdown but any thread that performs large memory scan - for example a worker thread that scans large amount of data might be a culprit as well.

For some customer applications when disabling HT we saw 10% increase in performance. So make sure that you do your home work before you decide to hyper on not to hyper :-)

Hope this information was useful. Let me know if you have any questions!

Be Aware: VirtualAlloc with MEM_RESERVE can fail even if you have plenty of VAS inside of the process

slavao — Fri, 03 Jun 2005 19:04:00 GMT

Many developers assume that if you have plenty of free VAS in your process VirtualAlloc (http://msdn.microsoft.com/library/default.asp?url=/library/en-us/memory/base/virtualalloc.asp) and VirtualAllocEx (http://msdn.microsoft.com/library/default.asp?url=/library/en-us/memory/base/virtualallocex.asp) calls with MEM_RESERVE parameter can't fail. This is not actually true. Remember that when allocating VAS region OS creates VAD (Virtual Address Descriptor). VAD manages VAS region, for more info read http://blogs.msdn.com/slavao/archive/2005/01/27/361678.aspx. When system is low on physical memory which includes RAM and swap file, VAD's allocation might fail. The failure will cause VirtualAlloc* call to fail. It means that if system low on physical memory VirtualAlloc* could fail. This becomes very important on 64 bit platforms. When running on 64 bit, developers might assume that reservation never fails - which is wrong.

Hope it was useful.

Be Aware: IA64 Stack Size

slavao — Sat, 19 Mar 2005 20:35:00 GMT

Stack subject in Windows OS is fascinating. There are so many interesting technical problems surrounding it. The moment you think you fully understand everything about stack you suddenly discover yet another mystery. And so it goes.

In the last couple of weeks I have been approached by our testers and support engineers asking about for how many threads they need to configure SQL Server on IA64 platform with 4GB of RAM. For example they asked is it ok to set number of threads to be 1024?

You probably know that SQL Server commits its stacks upfront. The reason we do it is to avoid unexplained server death - if there is no physical memory and machine is out of swap file, a process can disappear without leaving any traces. This will happen when a process will try to grow its stack. As any respectful server product SQL Server can't afford such behavior.

SQL Server on IA64 is configured to use 2MB of stack size. You can verify it by looking at SQL Server image header. Does it really consume that much of committed memory per thread? The answer is no. On IA64 there are two stacks - regular stack and backing store. Backing store is a spill space for cpu register engine. It is used by cpu to spill registers to memory when it no longer has registers available.

When a process specifies committed stack size to be 2MB, Windows commits 2MB for regular stack and 2MB for backing store. Consequently each thread will commit 4MB of RAM (If you only specify reservation Windows will reserve that much but not commit). Fig1 shows what the thread stack really looks like on IA64.

----------------------0x00000000000200000

| 2MB

| Stack

| / \

| |

----------------------0x00000000000400000

| 2MB

| Backing store

| |

| \ /

----------------------0x00000000000600000

Fig 1.

I would like to emphasize couple of interesting points. First Windows reserves a contiguous VAS region of size of 4MB for both stack and backing store. Second it divides this region in two so that the stack and the backing store grow in different directions.

Now we are in the position to answer the original question of how much physical memory 1024 threads will consume. The answer is 4MB*1024= 4GB. Considering that max server memory in SQL Server only reflects size of Buffer Pool and usually it is configured to be close to amount of physical memory on the box, this configuration might cause SQL server to crawl. This comes as a real surprise to many people. Max amount of threads SQL needs to be configured for does depend on the type of application running on top of server but in this case it definitely needs to be below 1024. Even 512 number of threads could be too much.

This post will be incomplete if I didn't somehow mention stack overflow. Backing store makes IA64 stack handling to be different from other platforms. Stack overflow is not an exception in this case - OS needs to handle stack overflow both for regular stack and backing store. Surprise! For you it means that you have to be really careful when recovering from stack overflow especially when you diced to restore guard page by hand. But this is a whole different story on its own.

I hope you learned something from this post. Comments are welcomed!

Have a great weekend!

Be aware: a new section in my blog

slavao — Tue, 22 Feb 2005 09:12:00 GMT

Periodically, I bump into unusual behavior, features, of compiler, operating system, strange design and coding decisions made by someone else that makes me wonder. After giving it some thought, I often come to the conclusion that behavior or observed specifics are justifiable. However if I knew it upfront, It would have helped me to avoid unnecessary mistakes and complexity.

Today I would like to open new section in my blog: Be Aware. In this section I would like to spend some time on this exact subject: strange behaviors and unexpected features. Sometimes these type of things are fascinating, sometimes just interesting and sometimes are just naïve. My goal is to point them out so that when you make your own decisions you will have more information to rely on. My goal is not to uncover bugs or start some kind of religious debates - I really don't like them. I strongly believe that problem drives a solution. What I mean by this is that something outrages in one place will be fully applicable in the other.

So let's begin :-).

The topic of variable's placement have been covered to full extend by now. Some developers prefer to declare all variables upfront some in the scope where variables get actually used. Personally, I have heard cons and pros for both sides and kind of agree with both. I think there is no black and white. In some cases you do have to declare a variable inside of the given scope. Recently I have observed interesting debugger's behavior while debugging dumps. As all of you know debugger usually has difficulties to show local variables for retail, optimized build. This is kind of expected in many cases. Before you can rely on a value, you have to confirm that the value is really correct. In windbg I usually use dv command to output local variables

> dv -V

Using the command with -V lists the actual placement of the variable on the stack. What I noticed is that windbg wasn't displaying variables declared in local scopes at all even if instruction pointer was inside of the scope. It just doesn't list them! So I usually end up chasing those variables by disassembling the code and following stack manipulations. (Knowing original stack assignment in this cases would really helped) The process works 100% of the time, but I would have saved hours if the code declared variables in the beginning of the function. At least the debugger would gave me a clue where the variable resides on the stack.

Probably the point here is that avoiding unnecessary complexity could enable tools to do better job and make everyone life easier. Remember tools, as any software, well tested for the common case. So next time when declaring a variable in an internal scope just stop for a second and think: Is it really necessary?

I have more stories around complexity but I will keep them until next time :-).

Look at Memory Leaks - Classify and Identify

slavao — Fri, 04 Feb 2005 20:16:00 GMT

In my previous post I talked about type of Memory Pressures. Today I would like to talk about related subject – Memory Leaks. Before you continue reading please make sure that you are familiar with how memory is managed on Windows.

Memory leaks are bugs that happen very often. I think it is very important to know how to approach the issue. Once you learned the process, these bugs are not as evil as everyone things.

When leaks occur, the first thing developer needs to do is to classify them. How many times you have heard people are just saying the application failed or became unresponsive due to memory leak? A phrase memory leak could mean leak of several memory resources. Remember in my previous posts I talked about VAS, physical memory, page file and etc. An application can leak any of these resources. This means that memory leak is incomplete phrase that doesn’t have much meaning. One needs to be very specific when referring to a leak.

Leak Classification is very important. The table below represents memory leak type and tools that could be used to classify them. Keep in mind that there could be more tools and more leak types. I just enumerated the most common types that I dealt with in the past.

Memory Resource/Tool	Task Manager	PerfMon	WinDbg	WinDbg – Local Kernel Debugger	VADUMP	SQL Server
VAS		Process/Virtual Bytes	!vadump !heap	!vad	X
Physical Memory	Virtual Memory	Process/Private Bytes	!vadump !heap	!memusage !vm	X
Page File	Commit Charge	Memory/Committed Bytes		!memusage !vm	X
AWE						Buffer Manager/Total pages
Handle	Handles	Objects	!handle	!handle

Once leak is classified, it needs to be identified. Identification of a leak involves a tracking of leaking stack. Once stack is known leak is identified. Keep in mind in many cases it is not sufficient to identify file line and number. Below table represents tools that could be used to identify a leak:

Memory Resource/Tool	umdh/pageheap	LeakDiag	WinDbg	WinDbg – Local Kernel Debugger
VAS		Virtual Allocator	!heap -l
Physical Memory / Page File	Windows Heaps	Windows Heaps CRT Heap COM Heap MDAC Heap	!vadump !heap	!memusage
AWE
Handle			Break points on CreateEvent, CreateMutex, … CloseHandle	!htrace

Usually classifying a leak is not hard and should be done very fast. You should be able to teach your testers, QA personal, to do it. Moreover the classsification process could be automated. Identifying a leak is usually harder and more time consuming process. It might involve multiple restarts, use of different tools and etc... Knowing how a tool works and being comfortable with it is the key. Getting there does require experience. Once you are there, dealing with memory leaks becomes easier process. The only problem is that there will be always a leak that your tools can’t handleJ. This is the case when you sit down and write your own LeakDiag.

There are several commercially available tools enabling you to classify and identify leaks. I haven’t looked at them for long time but from my past experience they don’t work as well as the ones I outlined in here. Before you decide to buy one make sure you understand how the tool works. If tool is capable of handling leaks only at process shutdown, it has very low value. Moreover, If you work on high-end server and have your own memory manager, most of those tools are not applicable.

I will be glad to hear your comments.

Enjoy weekend!

Getting an exception call stack from the catch block (C++)

slavao — Sun, 30 Jan 2005 23:36:00 GMT

This is my first post in this category so I am really looking forward to your comments :-) Today I would like to cover a specific feature related to our implementation of C++ exception handling. Every day I realize that not many people are familiar with this feature. When they learn about it they really get surprised :-). You need to see their reaction, it is just awesome! So now let’s see yours :-).

Have you ever had a day when you are starring at catch block and willing you knew the stack of original exception? Do you remember how frustrated you were :-)? Well it turns out that the original stack is there under your fingertips. The debugger just hides it . Did I get you going J?

MS implementation of C++ exception handling doesn't actually pop the stack until the flow control leaves catch block. From C++ standard "A throw-expression initializes a temporary object, the type of which is determined by removing any top-level cv-qualifiers from the static type..." It means that CRT needs to a make copy of an object somewhere when exception is thrown. As you can guess, it makes a copy of it on the stack! It means that by default run time can't pop the stack until exception is dismissed. This feels good J

Here is what actually happens on x86 platform when exception gets thrown. This is oversimplified view but you will get an idea:

- Copy of the object is created on the stack

- Appropriate handler, catch block, is found

- Runtime calls object destructors located on the stack

- EBP is adjusted to a frame containing the catch block

- ESP is NOT TOUCHED

- Flow of control transferred to catch block

- When done with a catch block the flow of control returns back to the place where it came before entering catch block

- ESP is adjusted

- Flow of control is transferred outside of catch block

Amazing right! Now since EBP gets adjusted before getting to the catch block debugger doesn’t show the real stack. It shows you stack as you would expect: without exception on the stack!

Lets take a look at the example:

int __cdecl main (int argc, char* rgArg [])

{

BYTE byte = 10;

try

{

throw (10);

}

catch (int exception)

{

BYTE* pStack;

__asm

{

mov pStack, esp;

}

printf ("Stack difference %d\n",&byte - pStack);

}

// To see real flow control we need to put some code

// here

printf (“I am almost done\n”);

return 0;

}

Output:

Stack difference 1595

Wow! Now let’s take a look at the stack. I put a break point in the debugger on the first printf statement:

000aff84 01001469 00000001 002d5cc0 002d3b30 test!main+0x44

000affc0 77e4f38c 00000000 00000000 7ffdf000 test!mainCRTStartup+0xb0

000afff0 00000000 010013b9 00000000 78746341 kernel32!BaseProcessStart+0x23

This is what you would expect right? Now let’s take a look at registers

0:000> r

eax=01001331 ebx=01001331 ecx=00000100 edx=00000002 esi=000afbc8 edi=000aff78

eip=01001334 esp=000af938 ebp=000aff84 iopl=0 nv up ei pl nz ac po nc

cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000216

test!main+0x44:

01001334 8d4def lea ecx,[ebp-0x11] ss:0023:000aff73=0aff580a

Please notice a big difference between ebp and esp! The real question now is how to get the real stack. The simple way is to make a search for 0001003f pattern. As many of you know this is usually how the first four bytes of CONTEXT look like on x86 platform

0:000> s -d @esp @ebp 0001003f

000afbe8 0001003f 00000000 00000000 00000000 ?...............

All we have to do know is to switch context:

0:000> .cxr 000afbe8

eax=000afeb8 ebx=7ffdf000 ecx=00000000 edx=002d3b30 esi=000aff48 edi=000aff48

eip=77e649d3 esp=000afeb4 ebp=000aff08 iopl=0 nv up ei pl nz na po nc

cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206

kernel32!RaiseException+0x51:

77e649d3 5e pop esi

000aff08 7c3929f8 e06d7363 00000001 00000003 kernel32!RaiseException+0x51

000aff48 01001331 000aff68 010018d4 00000000 MSVCR80!_CxxThrowException+0x34

000aff84 01001469 00000001 002d5cc0 002d3b30 test!main+0x41

000affc0 77e4f38c 00000000 00000000 7ffdf000 test!mainCRTStartup+0xb0

000afff0 00000000 010013b9 00000000 78746341 kernel32!BaseProcessStart+0x23

And here you go J. Also we could have used dps command to find a previous frame.

0:000> dps @esp

000af938 7c392a47 MSVCR80!_NLG_Return

000af93c 000aff78

000af940 000afbc8

000af944 000af954

000af948 000aff78

000af94c 01001331 test!main+0x41

000af950 000aff84

000af954 000af980

000af958 7c3929b5 MSVCR80!_CallCatchBlock2+0x4a

000af95c 01001331

000af960 000aff78

000af964 00000100

000af968 000af9d4

Now all we need to do is reset ebp to 000af954

0:000> r ebp=000af954

000af954 7c3929b5 01001331 000aff78 00000100 test!main+0x44

000af980 7c391bc4 000aff78 01001918 01001331 MSVCR80!_CallCatchBlock2+0x4a

000af9e4 7c392081 000afbc8 000aff78 00000001 MSVCR80!CallCatchBlock+0x84

000afa14 7c3923dd 000afbc8 000afbe8 000afba0 MSVCR80!CatchIt+0x5c

000afa70 7c3925c6 000afbc8 000aff78 000afbe8 MSVCR80!FindHandler+0x26e

000afaa4 7c392680 000afbc8 000aff78 MSVCR80!__InternalCxxFrameHandler+0xc5

000afae0 77f68cf6 000afbc8 000aff78 000afbe8 MSVCR80!__CxxFrameHandler3+0x26

000afae0 77f68cf6 000afbc8 000aff78 000afbe8 ntdll!ExecuteHandler2+0x26

000afb04 77f68cc5 000afbc8 000aff78 000afbe8 ntdll!ExecuteHandler2+0x26

000afbb0 77f45275 00000000 000afbe8 000afbc8 ntdll!ExecuteHandler+0x24

000afbb0 77e649d3 00000000 000afbe8 ntdll!KiUserExceptionDispatcher+0xe

000aff08 7c3929f8 e06d7363 00000001 00000003 kernel32!RaiseException+0x51

000aff48 01001331 000aff68 010018d4 00000000 MSVCR80!_CxxThrowException+0x34

000aff84 01001469 00000001 002d5cc0 002d3b30 test!main+0x41

000affc0 77e4f38c 00000000 00000000 7ffdf000 test!mainCRTStartup+0xb0

000afff0 00000000 010013b9 00000000 78746341 kernel32!BaseProcessStart+0x23

This stack dump is even better. It gives you full stack right before we executed jump to catch block!

The consequence of this behavior is that you need to be very careful when re-throwing exceptions from the catch block in recursive or deep functions - you might hit stack overflow! We actually did and not only once J.

As many people say you live you learn :-)

Enjoy the rest of the day!