I write below code to verify the behaviors of AddVectoredExceptionHandler, AddVectoredContinueHandler and SetUnhandledExceptionFilter:

#include "stdafx.h"

LONG WINAPI MyVectorContinueHandler(PEXCEPTION_POINTERS p)
{
 printf("in my vectored continue handler\r\n");
 return EXCEPTION_CONTINUE_SEARCH;
}

LONG WINAPI MyVectorExceptionFilter(PEXCEPTION_POINTERS p)
{
 printf("in my vectored exception filter\r\n");
 return EXCEPTION_CONTINUE_SEARCH;
}

LONG WINAPI MyUnhandledExceptionFilter(PEXCEPTION_POINTERS p)
{
 printf("in my unhandled excepiton filter\r\n");
 return EXCEPTION_CONTINUE_SEARCH;
}

LONG MyExceptFilter()
{
 printf("in my filter\r\n");
 return EXCEPTION_CONTINUE_SEARCH;
}

int _tmain(int argc, _TCHAR* argv[])
{
 LPTOP_LEVEL_EXCEPTION_FILTER pOriginalFilter = SetUnhandledExceptionFilter(MyUnhandledExceptionFilter);
 
 AddVectoredExceptionHandler(1,MyVectorExceptionFilter);
 AddVectoredContinueHandler(1,MyVectorContinueHandler);

 __try
 {
  //trigger an access violation here
  int* p;
  p = 0;
  *p = 10;
 }
 __except(MyExceptFilter())
 {
  printf("in my handler\r\n");
  printf("exception code: 0x%x\r\n",GetExceptionCode());
 } 
}

Build and then run the code on Windows 2003, get below result which is as expected:

in my vectored exception filter
in my filter
in my unhandled excepiton filter
in my vectored continue handler

We can see that:

1. The vectored exception filter (MyVectorExceptionFilter) is called before the frame based SEH filter.

2. MyUnhandledExceptionFilter gets called since the exception is not handled (EXCEPTION_CONTINUE_SEARCH is returned from previous filters).

3. Finally, the vectored continue handler (MyVectorContinueHandler) gets called.

However, runs the same code on Windows 7, we get below result:

in my vectored exception filter
in my filter
in my unhandled excepiton filter

then Windows Error Reporting gets involved and below window pops up, the vectored continue handler is not called:

Another thing to note: we know that MyUnhandledExceptionFilter is called by the kernel32!UnhandledExceptionFilter:

 ChildEBP RetAddr 
0020f880 77ec2c4a test!MyUnhandledExceptionFilter+0x1e
0020f908 77d85a74 kernel32!UnhandledExceptionFilter+0x127
0020f910 77d2d950 ntdll!__RtlUserThreadStart+0x62
0020f924 77d2d7ec ntdll!_EH4_CallFilterFunc+0x12
0020f94c 77d565f9 ntdll!_except_handler4+0x8e
0020f970 77d565cb ntdll!ExecuteHandler2+0x26
0020fa20 77d56457 ntdll!ExecuteHandler+0x24
0020fa20 00000000 ntdll!KiUserExceptionDispatcher+0xf

If you debug the code in Visual Studio (or run it under other debugger such as windbg), if you set a break point in test!MyUnhandledExceptionFilter and you will find the break point cannot be hit. This is because in kernel32!UnhandledExceptionFilter, it detects if the code is runs under a debugger if so, it just returns EXCEPTION_CONTINUE_SEARCH and the customized unhandled exception filter (test!MyUnhandledExceptionFilter in this case) is not called at all.