Software Sleuthing : C/C++

MiniDumps and "Bad" Stacks

joshpoley — Mon, 10 Nov 2008 19:07:00 GMT

A fellow reader sent a comment outlining the following problem:

I'm using the DMPSTK example from the Debugging Tools SDK. If I generate a crash dump from within Visual Studio 2005, I can see the call stack perfectly. However, if I create my own crash dump (the same as you do in your blog), I get several LoadLibrary(ext), LoadLibrary(exts), ... failures. What's more important to notice is that the callstacks are different.

The LoadLibrary messages are benign and just an artifact of the debugger attempting to load additional plug-ins. But with regard to the stack traces, things get more interesting.

It turns out that the crash dump files created with Visual Studio are slightly different from the ones created via the API MiniDumpWriteDump. On top of that, WinDBG and the debugging engine exposed by WinDBG handle things a little bit differently with regard these dumps. The main difference stems from the use of embedded "contexts". A context contains the basic state of the system (CPU registers for example), and crash dumps can contain an embedded context which gets added in if you pass down the appropriate exception details to MiniDumpWriteDump.

If you were to load up one of these API generated crash dump files in WinDBG itself, the program will kindly tell you that there is an additional context and in order to use it, you need to issue the ".ecxr" command. Until you do this, the stack trace you will see possibly is not the stack you are interested in. This is where things are going wrong for our reader, who was displaying the default stack via the code in DMPSTK sample.

Luckily for us, we can do the equivalent to .ecxr in our code allowing us to get the "correct" stack.

To start with we replace IDebugControl for IDebugControl4. This then gives us access to the GetStoredEventInformation and GetContextStackTrace methods.

If we were to modify the initial code from this blog entry, we would end up with something like this which lets us try to get the embedded context, otherwise just fall back on the old standby (GetStackTrace).

HRESULT DumpStack(IDebugControl4 *control, IDebugSymbols *symbols)

{

#define MAX_FRAMES 1024

HRESULT hr = S_OK;

DEBUG_STACK_FRAME stackFrames[MAX_FRAMES] = {0};

ULONG numFrames = 0;

char context[1024] = {0};

ULONG type = 0;

ULONG procID = 0;

ULONG threadID = 0;

ULONG contextSize = 0;

char *contextData = NULL;

// look for an embedded event

hr = control->GetStoredEventInformation(&type, &procID, &threadID,

context, sizeof(context), &contextSize, NULL, 0, 0);

// get the stack trace

if(SUCCEEDED(hr))

{

contextData = new char[MAX_FRAMES*contextSize];

hr = control->GetContextStackTrace(context, contextSize,

stackFrames, ARRAYSIZE(stackFrames), contextData,

MAX_FRAMES*contextSize, contextSize, &numFrames);

}

else

{

hr = control->GetStackTrace(0, 0, 0, stackFrames,

ARRAYSIZE(stackFrames), &numFrames);

}

...

Just don't forget to delete the newly allocated "contextData" buffer at the end of the function.

Developer Tip: Add a blank macro so testers can inject code

joshpoley — Mon, 11 Aug 2008 19:39:00 GMT

While I was working on writing some API tests for a module that was being integrated into the Xbox Dashboard application, there was some rework to the class that happened in the middle process. In doing so, the developer decided to make a bunch of the functionality private, causing my existing test to fail to build (as they no longer had access to anything). Luckily the developer left me a little cookie in his code:

#ifndef CONNECTION_POOL_TEST_FRIENDS

#define CONNECTION_POOL_TEST_FRIENDS()

#endif

class CDdlConnectionPool

{

...

CONNECTION_POOL_TEST_FRIENDS();

};

By providing a blank macro inside the class, he gave me the opportunity to modify the class without having to change the header file. I just have to define the CONNECTION_POOL_TEST_FRIENDS macro from my test code before including the above header file, and I can inject my own friendliness:

// enable the API test to be a "friend" of the connection pool

#define CONNECTION_POOL_TEST_FRIENDS \

friend HRESULT Pool_InitializeTest(void); \

friend HRESULT Pool_AddEntryTest(void);

#include <dashdl.h>

I like cookies.

Leave well enough alone

joshpoley — Thu, 17 Jul 2008 17:53:00 GMT

I was reviewing a check-in the other day, and one of the changes was to fix a line of code which resulted in one of the new "safe CRT" warnings:

warning C4996: 'strcpy': This function or variable may be unsafe. Consider using strcpy_s instead.

Here is the original code:

HRESULT GetDefaultList(char** tests)

{

if(!tests) return E_INVALIDARG;

char *testList = "item1, item2, item3";

*tests = new(std::nothrow) char[strlen(testList) + 1];

if(!*tests) return E_OUTOFMEMORY;

strcpy(*tests, testList);

return S_OK;

}

Here is the updated code that was checked in:

HRESULT GetDefaultList(char** tests)

{

if(!tests) return E_INVALIDARG;

char *testList = "item1, item2, item3";

*tests = new(std::nothrow) char[strlen(testList) + 1];

if(!*tests) return E_OUTOFMEMORY;

strcpy_s(*tests, strlen(*tests), testList);

return S_OK;

}

The problem is that we took perfectly working and valid code, and because of a compiler warning, we introduced a run-time bug: strlen of uninitialized data. The obvious take away from this is to ignore the compiler, but a better approach is to actually think about what you are doing.

1. Think about and review every line of code you write. Even (especially?) if it is just a "one line fix".

2. Introducing code churn always introduces risk. If your code isn't broken to begin with, "fixing" various warnings (either from a compiler or static analysis tool) might not be the best use of everybody's time.

I'm a big proponent of getting projects to build "clean" on the most restrictive warning level. But when tools encourage you to iterate over valid code, it might be better to just disable the warnings over a specific block of code.

Automating Crash Dump Analysis: Some Final Thoughts

joshpoley — Mon, 23 Jun 2008 19:04:00 GMT

For a reference, here are some links to the previous parts in this series:

· Prolific Usage of MiniDumpWriteDump

· Opening a Crash Dump File

· Getting the Stack from a .DMP File

· Getting the Crash Details from a .DMP File

· Showing the Disassembly from a .DMP File

So now the question that begs to be answered is "how do we best leverage all this information?" Let's start by assuming we can stick all of it into a database for further (and easier) querying. This then opens up a whole slew of natural reports, such as, how often do we see similar types of crashes, or which functions or modules in our code base are the scariest.

If we add the ability to talk to a source control system, then we could start tying crashes to owners or developers based on the source control history (since we know the filenames and line numbers from the stack trace). This then gives us the ability to send out notification emails ("You have a new crash dump to look at") or to assign associated bugs to the correct individuals.

And speaking of bugs: If we are linked into the bug tracking system, we can then add support for some more automatic maintenance activities. For example, when a new crash dump comes in, we can look in our database and see if it is identical to something we've seen before. If it is, and any associated bugs are still active, we may just want to note the occurrence and not waste anybody's time by manually debugging the crash (we could automatically edit the bug to indicate that a new dump is available if needed). On the other hand, if the bug has already been "fixed", then we will want somebody to pay attention to the new crash dump.

Showing the Disassembly from a .DMP File (Automating Crash Dump Analysis Part 4)

joshpoley — Tue, 17 Jun 2008 23:31:00 GMT

When investigating a crash, it can often be beneficial to see the assembly leading up too, and even following the event. It turns out that generating the corresponding assembly at an address is pretty easy. The hard part is finding a good starting point for enumerating if we want to look backwards from a given address. On machine architectures that use a variable length instruction set, we can't necessarily walk backwards and expect to get the correct instruction listing, so we need to jump backwards to a know good location, then fast-forward to where we want to be.

This function will take the IDebugControl and IDebugSymbols interface pointers opened earlier, then dump out 'numLines' of assembly instructions preceding and following the crash at 'addr'. The input address can be taken from the stack frame or the exception record in the event information.

HRESULT DumpDisassembly(IDebugControl *control, IDebugSymbols *symbols,

ULONG64 addr, int numLines)

{

HRESULT hr = S_OK;

char asmBuffer[1024] = {0};

ULONG asmSize = 0;

ULONG64 endAddr = 0;

ULONG64 ip = addr;

ULONG64 funcStart = 0;

ULONG64 offset = 0;

ULONG procType = 0;

bool fixedInstructionLength = false;

printf("\nDisassembly\n");

// test for an invalid instruction pointer

if(addr == 0)

{

printf(" no data at address 0x00000000\n");

return hr;

}

// determine the algorithm to use

hr = control->GetEffectiveProcessorType(&procType);

if(SUCCEEDED(hr))

{

if(procType == IMAGE_FILE_MACHINE_ARM)

fixedInstructionLength = true;

// insert additional fixed length instruction

// platforms here

}

// locate the start of the function

symbols->GetNameByOffset(addr, NULL, 0, NULL, &offset);

funcStart = addr - offset;

// find the beginning address

if(fixedInstructionLength)

{

// if we are using a fixed length instruction set

// then we can easily step backward n lines

ip = addr;

hr = control->GetNearInstruction(addr, -numLines, &ip);

}

else

{

// on a variable length instruction architecture, we need

// to start at a known good location (funcStart in this case).

// Then walk each instruction until we get to the addr,

// keeping track of the history in a circular buffer.

ULONG64 *circBuffer = new(std::nothrow) ULONG64[numLines];

size_t index = 0;

if(circBuffer)

{

// initialize the buffer

for(int i=0; i<numLines; i++)

circBuffer[i] = 0;

// make sure there is at least one entry

circBuffer[0] = ip;

// walk from the start till the addr

ip = funcStart;

while(ip < addr)

{

circBuffer[index] = ip;

hr = control->GetNearInstruction(ip, 1, &ip);

if(FAILED(hr)) ++ip;

index = (index+1) % numLines;

}

// find the oldest (non-zero) entry

{

ip = circBuffer[index];

index = (index+1) % numLines;

} while(ip == 0);

delete[] circBuffer;

}

// make sure we don't go past the beginning of the funciton

if(ip < funcStart) ip = funcStart;

// dump out the instructions before and after the event

for(int lines=0; lines<numLines; )

{

hr = control->Disassemble(ip, DEBUG_DISASM_EFFECTIVE_ADDRESS,

asmBuffer, ARRAYSIZE(asmBuffer)-1, &asmSize, &endAddr);

if(FAILED(hr))

break;

if(ip == addr)

printf("> %s", asmBuffer);

else

printf(" %s", asmBuffer);

if(ip > addr) ++lines;

ip = endAddr;

}

return hr;

}

MSDN References

· GetEffectiveProcessorType

· GetNameByOffset

· GetNearInstruction

· Disassemble

Getting the Crash Details from a .DMP File (Automating Crash Dump Analysis Part 3)

joshpoley — Fri, 06 Jun 2008 20:48:00 GMT

Aside from a stack trace (the "where"), you probably want to know something about the crash (the "why" or the "how"). With the code below, we can query some of the basic details about what happened. From this we will at least be able to determine if the application attempted to write to a protected page, or if it decided to feel the effects of a division by zero.

HRESULT DumpEvent(IDebugControl *control, IDebugSymbols *symbols)

{

union ExtraInfo

{

DEBUG_LAST_EVENT_INFO_EXCEPTION exceptionInfo;

// as needed, we can add more of the

// DEBUG_LAST_EVENT_INFO_xyz structs here

};

HRESULT hr = S_OK;

ULONG type = 0;

ULONG procID = 0;

ULONG threadID = 0;

ExtraInfo extraInfo;

ULONG extraInfoUsed = 0;

char description[80] = {0};

ULONG bugCheckCode = 0;

ULONG64 bugCheckArgs[4] = {0};

ZeroMemory(&extraInfo, sizeof(extraInfo));

printf("\nFault Information\n");

// look for Bug Check data

hr = control->ReadBugCheckData(&bugCheckCode, &bugCheckArgs[0], &bugCheckArgs[1],

&bugCheckArgs[2], &bugCheckArgs[3]);

if(SUCCEEDED(hr))

{

printf(" Bug Check: %X (0x%08X, 0x%08X, 0x%08X, 0x%08X)\n",

bugCheckCode, bugCheckArgs[0], bugCheckArgs[1],

bugCheckArgs[2], bugCheckArgs[3]);

}

// get the fault information

hr = control->GetLastEventInformation(&type, &procID, &threadID,

&extraInfo, sizeof(extraInfo), &extraInfoUsed, description,

ARRAYSIZE(description)-1, NULL);

if(FAILED(hr))

goto cleanup;

printf(" Description: %s\n", description);

// if we hit an exception, and we understand the type of exception, write

// out some additional information

if((type == DEBUG_EVENT_EXCEPTION) &&

(extraInfoUsed >= sizeof(extraInfo.exceptionInfo)))

{

EXCEPTION_RECORD64 *er = &extraInfo.exceptionInfo.ExceptionRecord;

printf(" Type: %s\n", GetExceptionName(er->ExceptionCode));

if(er->ExceptionCode == EXCEPTION_ACCESS_VIOLATION ||

er->ExceptionCode == EXCEPTION_IN_PAGE_ERROR)

{

if(er->ExceptionInformation[0] == 0)

printf(" Read at: 0x%08X\n", er->ExceptionInformation[1]);

else if(er->ExceptionInformation[0] == 1)

printf(" Write at: 0x%08X\n", er->ExceptionInformation[1]);

else if(er->ExceptionInformation[0] == 8)

printf(" User Mode Fault: 0x%08X\n", er->ExceptionInformation[1]);

}

cleanup:

return hr;

}

The GetExceptionName function is a simple helper that you can provide, which just transforms the exception defines into human readable strings.

MSDN References

· ReadBugCheckData

· GetLastEventInformation

· EXCEPTION_RECORD

Getting the Stack from a .DMP File (Automating Crash Dump Analysis Part 2)

joshpoley — Mon, 02 Jun 2008 18:06:00 GMT

So now that we have a memory dump file, and know how to open it, we will want to pull some useful data out. To start with, we will grab the stack trace (which is arguably one of the more important pieces). And in case you've stumbled across the "dumpstk" sample in the WinDBG install\sdk directory, we will do things a little bit differently and not call OutputStackTrace() for these reasons:

1. By enumerating the stack structures directly, we have more control over how to format the results.

2. We don't have to deal with supplying an IDebugOutputCallbacks implementation (see the out.cpp file which accompanies the "dumpstk" sample).

3. We have individual control over each frame (OutputStackTrack can "chunk" up frames as they are sent to the IDebugOutputCallback interface).

Here is a sample function that takes pointers to the IDebugControl and IDebugSymbols interfaces (which are opened in part 1) and will dump the stack trace with some good old fashioned printf statements.

HRESULT DumpStack(IDebugControl *control, IDebugSymbols *symbols)

{

HRESULT hr = S_OK;

DEBUG_STACK_FRAME stackFrames[4096] = {0};

ULONG numFrames = 0;

// get the stack trace

hr = control->GetStackTrace(0, 0, 0, stackFrames, ARRAYSIZE(stackFrames),

&numFrames);

if(FAILED(hr))

goto cleanup;

printf("\nStack Trace\n");

// dump out the details for each function call

for(ULONG frame=0; frame<numFrames; frame++)

{

HRESULT symhr;

char name[512];

unsigned __int64 offset = 0;

ULONG lineNo = 0;

// get the module, function, and offset

ZeroMemory(name, ARRAYSIZE(name));

symhr = symbols->GetNameByOffset(stackFrames[frame].InstructionOffset,

name, ARRAYSIZE(name)-1, NULL, &offset);

if(SUCCEEDED(symhr))

printf("%s+0x%I64X", name, offset);

else

printf("0x%08I64X", stackFrames[frame].InstructionOffset);

// look for source file name and line number

ZeroMemory(name, ARRAYSIZE(name));

symhr = symbols->GetLineByOffset(stackFrames[frame].InstructionOffset,

&lineNo, name, ARRAYSIZE(name)-1, NULL, NULL);

if(SUCCEEDED(symhr))

printf("\n %s(%u)", name, lineNo);

printf("\n");

}

cleanup:

return hr;

}

Everything here is pretty straight forward: we retrieve the stack frame data, then just walk through each element of the populated array and pull out the module, the function name, and if we have full symbols, the filename and line number of the source code.

One thing you might want to add here is some intelligence about detecting recursive sequences. If a stack trace is extremely long and filled with repeating entries, it might be better to just write out the top and bottom of the stack and omit the majority of the redundant line items.

MSDN References

Opening a Crash Dump File (Automating Crash Dump Analysis Part 1)

joshpoley — Tue, 27 May 2008 19:20:00 GMT

So let's assume for the moment that you have a collection of crash dump files from your team's application. These files may be generated from your stress tests, betas, etc. But where they come from really isn't important, what is important is that we want some way to dig into these files and gather information, evaluate priority, etc., without having to manually open each one in the debugger.

Once you install WinDBG, you are given a COM interface allowing you to integrate debugging functionality into your own application. We can then use this to drive the task of opening and querying a crash dump file. When installing, just make sure you select the SDK option, which will give you the header files you will need. To make you life happier, you should also install the symbol packages for all the Operating Systems you support (also available from the above link) - doing so will give you function names from system binaries in the stack traces we will eventually create.

To start with we will go over the minimum code to initialize the debugging library and open a crash dump file.

#include <windows.h>

#include "dbgeng.h" // windbg header

int __cdecl main(int argc, char* argv[])

{

HRESULT hr = E_FAIL;

IDebugClient *client = NULL;

IDebugControl *control = NULL;

IDebugSymbols *symbols = NULL;

// Initialize COM

hr = CoInitialize(NULL);

if(FAILED(hr))

goto cleanup;

// Create the base IDebugClient object

hr = DebugCreate(__uuidof(IDebugClient), (LPVOID*)&client);

if(FAILED(hr))

goto cleanup;

// from the base, create the Control and Symbols objects

hr = client->QueryInterface(__uuidof(IDebugControl), (LPVOID*)&control);

if(FAILED(hr))

goto cleanup;

hr = client->QueryInterface(__uuidof(IDebugSymbols), (LPVOID*)&symbols);

if(FAILED(hr))

goto cleanup;

// we can supplement the _NT_SYMBOL_PATH environment variable

// by adding a path here

symbols->SetSymbolPath("c:\\myApp\\release");

// the debugger will need to look at the actual binaries

// so provide the path to the exsecutable files

symbols->SetImagePath("c:\\myApp\\release");

// open the crash dump

hr = client->OpenDumpFile("c:\\myApp\\myApp.dmp");

if(FAILED(hr))

goto cleanup;

// wait for the engine to finish processing

control->WaitForEvent(DEBUG_WAIT_DEFAULT, INFINITE);

cleanup:

// cleanup and destroy the objects

if(symbols)

{

symbols->Release();

symbols = NULL;

}

if(control)

{

control->Release();

control = NULL;

}

if(client)

{

client->Release();

client = NULL;

}

// cleanup COM

CoUninitialize();

return 0;

}

The semicolon separated path you pass into SetSymbolPath() should provide the location of the .pdb symbol files for all binaries consumed by your application. The same goes for the paths to the actual binary files themselves, which is passed in via SetImagePath(). Without the symbols or the actual binaries, your investigations won't be as successful.

MSDN References

· DebugCreate() which is used to instantiate the initial IDebugClient interface.

· IDebugClient

· IDebugControl

· IDebugSymbols

Prolific Usage of MiniDumpWriteDump (Automating Crash Dump Analysis Part 0)

joshpoley — Mon, 19 May 2008 18:27:00 GMT

As a precursor to a series on programmatically querying crash dump files, I wanted to first talk about the MiniDumpWriteDump function.

You don't necessarily need to wait for your application to crash in order to generate a memory dump. As soon as you detect something bad happening, you can call MiniDumpWriteDump yourself and generate a crash dump file on the fly. If your application likes to eat exceptions or ignore error codes, or important asserts fire, or special test cases fail, then this is a great place to start collecting failure data for offline analysis.

And to make it super easy for you, I've created a helper function which will generate a filename based on the computer's name along with the date and time, then invoke the crash dump generator.

#include <dbghelp.h>

HRESULT GenerateCrashDump(MINIDUMP_TYPE flags, EXCEPTION_POINTERS *seh=NULL)

{

HRESULT error = S_OK;

// get the time

SYSTEMTIME sysTime = {0};

GetSystemTime(&sysTime);

// get the computer name

char compName[MAX_COMPUTERNAME_LENGTH + 1] = {0};

DWORD compNameLen = ARRAYSIZE(compName);

GetComputerNameA(compName, &compNameLen);

// build the filename: APPNAME_COMPUTERNAME_DATE_TIME.DMP

char path[MAX_PATH] = {0};

sprintf_s(path, ARRAYSIZE(path),

"c:\\myapp_%s_%04u-%02u-%02u_%02u-%02u-%02u.dmp",

compName, sysTime.wYear, sysTime.wMonth, sysTime.wDay,

sysTime.wHour, sysTime.wMinute, sysTime.wSecond);

// open the file

HANDLE hFile = CreateFileA(path,

GENERIC_READ|GENERIC_WRITE,

FILE_SHARE_DELETE|FILE_SHARE_READ|FILE_SHARE_WRITE,

NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);

if(hFile == INVALID_HANDLE_VALUE)

{

error = GetLastError();

error = HRESULT_FROM_WIN32(error);

return error;

}

// get the process information

HANDLE hProc = GetCurrentProcess();

DWORD procID = GetProcessId(hProc);

// if we have SEH info, package it up

MINIDUMP_EXCEPTION_INFORMATION sehInfo = {0};

MINIDUMP_EXCEPTION_INFORMATION *sehPtr = NULL;

if(seh)

{

sehInfo.ThreadId = GetCurrentThreadId();

sehInfo.ExceptionPointers = seh;

sehInfo.ClientPointers = FALSE;

sehPtr = &sehInfo;

}

// generate the crash dump

BOOL result = MiniDumpWriteDump(hProc, procID, hFile,

flags, sehPtr, NULL, NULL);

if(!result)

{

error = (HRESULT)GetLastError(); // already an HRESULT

}

// close the file

CloseHandle(hFile);

return error;

}

Sample call:

GenerateCrashDump((MINIDUMP_TYPE)

(MiniDumpNormal |

MiniDumpWithHandleData |

MiniDumpWithUnloadedModules),

NULL);

If you happen to have OS exception information available (via __try/__except), then you will definitely want to pass in the EXCEPTION_POINTERS information. This will embed important and detailed information about the exception into the crash dump.

And if you want to get really fancy, you can embed additional (custom) information via the 2^nd to last parameter to MiniDumpWriteDump. For example, if your application is consuming external documents or information, it can be very handy to have this information (or metadata about it) accessible when diagnosing crash dumps after the fact.

StrCmpLogicalA

joshpoley — Mon, 28 Apr 2008 21:41:00 GMT

Due to a policy around supporting and encouraging internationalizable software, there are certain system APIs which only have a UNICODE version (functions typically with the W postfix). One example which is missing the plain old ANSI version is the StrCmpLogicalW function. Unfortunately for me, I was in need of an A version. "Fixing" my code to use the W version was impractical given the data I had to work with, but I really liked the idea of having my strings sorted with numbers being placed in an order which made sense to a human being. So I did the only logical thing left to do...

Just to get my code up and running, I did the easy thing which was to write an A version which takes the ANSI strings as inputs, allocates two new buffers, converts the inputs into UNICODE, and then calls the W version. Needless to say this incurs a lot of overhead, as all this has to happen for each string compare, which isn't pretty.

My next cut (shown below) actually walks the ANSI strings and does some nitty-gritty comparisons.

int StrCmpLogicalA(const char *psz1, const char *psz2)

{

// handle NULL inputs

if(!psz1 && !psz2) return 0;

if(!psz1) return -1;

if(!psz2) return 1;

while(*psz1 && *psz2)

{

if(*psz1 >= '0' && *psz1 <= '9' &&

*psz2 >= '0' && *psz2 <= '9') // numerical

{

// keep track of where we are starting

const char* digit1 = psz1;

const char* digit2 = psz2;

// strip off any leading zeros

size_t leading1 = 0;

size_t leading2 = 0;

while(*digit1 == '0')

{

++leading1;

++digit1;

}

while(*digit2 == '0')

{

++leading2;

++digit2;

}

// scan to the end of the digits

while(*psz1 >= '0' && *psz1 <= '9')

++psz1;

while(*psz2 >= '0' && *psz2 <= '9')

++psz2;

// calc the number of digits

size_t len1 = psz1 - digit1;

size_t len2 = psz2 - digit2;

if(len1 < len2) return -1;

if(len1 > len2) return 1;

// now start walking over the digits

while(digit1 < psz1 && digit2 < psz2)

{

// test the number

if(*digit1 < *digit2) return -1;

if(*digit1 > *digit2) return 1;

++digit1;

++digit2;

}

// if we reach here, the numbers are the same, and

// psz1 and psz2 already point at the next character

// to test

// since we kept track of leading digits, we can add

// precedence based off that.

if(leading1 < leading2) return -1;

if(leading1 > leading2) return 1;

}

else // mixed and non numerical

{

unsigned char c1 = *psz1;

unsigned char c2 = *psz2;

// strip off the lower case bits

if(c1 >= 'a' && c1 <= 'z') c1 &= 0xDF;

if(c2 >= 'a' && c2 <= 'z') c2 &= 0xDF;

// test the characters

if(c1 < c2) return -1;

if(c1 > c2) return 1;

// else they are the same, keep walking

++psz1;

++psz2;

}

// check for unprocessed characters

if(!*psz1 && *psz2) return -1;

if(*psz1 && !*psz2) return 1;

// strings are equivalent

return 0;

}

As you can see this version does a case insensitive compare, if you happen to need one that does a case sensitive compare, then you just need to comment out the two lines of code which strip out the 6^th bit. If you want "abc03def" to be treated the same as "abc003def", then comment out the two lines where we test the leading digit count.

Disclaimers

1. This has only gone through minimal testing, use at your own risk, etc. (Though if you do find bugs let me know!)

2. Even though I named it StrCmpLogicalA in this blog, I make no implied statement about behavioral conformance with StrCmpLogicalW. I've never seen the algorithm or code behind the official API, and the above sample undoubtedly does things differently. In fact, I know it does some things differently.

3. Developers are encouraged to use UNICODE and my apologies go out to those trying to facilitate its wider-spread usage.

NULLize the throwing new

joshpoley — Wed, 26 Mar 2008 20:39:00 GMT

Old versions of Microsoft's C++ compiler supplied a form of the new keyword which would return NULL if the memory allocation failed (to mimic the behavior of malloc). The newer (more C++ compliant) versions of the compiler will instead throw a std::bad_alloc exception on failure. This change has lead to a frequent question on the internal mailing lists: "how do I get a version of new which returns NULL?" (¹)

There are a handful of reasons to ask this, the most common is that someone is dealing with a legacy codebase which wasn't designed to be exception friendly.

With Visual Studio 2008, there are two answers (²): the surgical approach, or the brute force approach.

Surgical

The surgical method allows a developer to go into individual call sites and specify whether the call to new should return NULL instead of throwing an error. This is accomplished using a placement new which takes an object from the std namespace, handily called "nothrow".

char *s = new(std::nothrow) char[123];

if(s == NULL) ...

While this approach does take a certain amount of grunt work to convert a large code base, this is the best approach as you can selectively decide which calls to new return NULL or which ones remain C++ standard friendly.

Brute Force

The sledge hammer approach is really pretty simple: you just need to link nothrownew.obj into your project and all calls to the default new operator will be returned to the old NULL returning version. The drawback is that all calls are changed, so if anybody is relying on the C++ conformant version, they will not work correctly when an error does happen. For example, any code which uses the Standard Template Library will be very unhappy with the results of doing this.

In case my choice of words hasn't conveyed the appropriate message yet, let me be utterly clear: It is highly recommended that you do not link with nothrownew.obj. Instead, you should be using the (std::nothrow) option to explicitly state which calls to new should return NULL.

For Testers

As a tester, it is important to know about the difference here so you can spot occurrences where the code you are testing (or even writing) is doing the wrong thing.

Footnotes

(1) The fact that this topic keeps coming up means that having more information about it is a good thing.

(2) In previous versions there were more than two answers, but there is no need to muddy the waters here now that things are simpler. If you are still running VS 2003 then take a look here (vs. the same documentation for 2008).

Writing Strings with OLE (CCommand)

joshpoley — Tue, 18 Mar 2008 23:38:00 GMT

This last weekend I was writing a COM object for submitting/editing entries into a SQL Server 2005 database we have for collecting project metrics. To do so, I was using the templated OLE DB functions: CCommand<CDynamicAccessor>, and things were progressing well until I needed to write a string value into a varchar field.

The SetValue Problem

The documentation for CDynamicAccessor's SetValue API was helpful enough to tell me that "special handling" is required when trying to store string data, but is a little vague on the specifics. After some searching around I came across a KB article (KB201390) which goes into a recommended solution which involves the horribly evil practice of blindly copying data into a buffer you don’t own. I am actually very surprised that this is the best they could come up with at the time, as this is a very good way to corrupt memory or quite possibly introduce a security vulnerability.

The Solution

After a little more poking around, I ran across the CDynamicStringAccessor, which has a nice SetString method. This let me avoid the evil hack suggested in the KB article, and did so with very little churn to my code. So now I was able to edit the strings in the database, and everything was happy with my testing. At least until the point where it didn't work...

The NULL Problem

I have a little test script for trying out my COM object which consists of a couple of lines, one of which calls my method for modifying an entry in the database, and one to read it back and display it. When I ran this test on a specific record, the update didn't stick: no matter what I tried to write, it always came back to my script as an empty string. And the utterly evil part was that no errors were generated. It turns out that if the field contained an actual NULL value (not an empty string) then the update would silently fail.

The Workaround

Since I own the database I'm trying to write to, I have the flexibility to go in and disallow NULL values on the fields I want to modify from OLE. When setting up this constraint, I also needed to provide an empty string as a default value (using two single quotes: '').

The Code

For the people (like myself) that prefer to see working samples, here is the code I used.

The SQL statement to create the table I will be writing too, notice the NOT NULL and DEFAULT values.

CREATE TABLE [Metrics](

[MetricID] [uniqueidentifier] NOT NULL CONSTRAINT [DF_Metrics_MetricID] DEFAULT (newid()),

[Name] [varchar](64) NOT NULL CONSTRAINT [DF_Metrics_Name] DEFAULT (''),

[Contact] [varchar](32) NOT NULL CONSTRAINT [DF_Metrics_Contact] DEFAULT (''),

[Units] [varchar](16) NOT NULL CONSTRAINT [DF_Metrics_Units] DEFAULT (''),

[Description] [text] NOT NULL CONSTRAINT [DF_Metrics_Description] DEFAULT (''),

[CustomURL] [varchar](256) NOT NULL CONSTRAINT [DF_Metrics_CustomURL] DEFAULT (''),

CONSTRAINT [PK_Metrics] PRIMARY KEY CLUSTERED

(

[MetricID] ASC

)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]

) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]

The code to open the database:

lastError = dbSource.OpenFromInitializationString(L"DRIVER={SQL Server};SERVER=myserver;"

L"DATABASE=mydatabase;UID=username;PWD=password;");

if(FAILED(lastError))

{

return lastError;

}

lastError = dbSession.Open(dbSource);

if(FAILED(lastError))

{

dbSource.Close();

return lastError;

}

Here is the code I used to open and edit the record.

CCommand<CDynamicStringAccessorW> recordset;

CDBPropSet propset(DBPROPSET_ROWSET);

propset.AddProperty(DBPROP_IRowsetChange, true);

propset.AddProperty(DBPROP_IRowsetUpdate, true);

propset.AddProperty(DBPROP_UPDATABILITY, DBPROPVAL_UP_CHANGE |

DBPROPVAL_UP_INSERT | DBPROPVAL_UP_DELETE);

// the MetricID is hard coded here for sample purposes

char* sql = L"SELECT [Contact] FROM Metrics WHERE [MetricID] = "

L"'{00000000-0000-0000-0000-000000000000}'";

lastError = recordset.Open(dbSession, sql, &propset);

if(FAILED(lastError))

{

goto cleanup;

}

lastError = recordset.MoveFirst();

if(FAILED(lastError))

goto cleanup;

if(lastError == DB_S_ENDOFROWSET) // row not found

{

lastError = 0x80000000 | DB_S_NORESULT;

goto cleanup;

}

lastError = recordset.SetString(1, newVal);

if(FAILED(lastError))

goto cleanup;

lastError = recordset.SetData();

if(FAILED(lastError))

goto cleanup;

lastError = recordset.Update();

if(FAILED(lastError))

goto cleanup;

cleanup:

recordset.Close();

return lastError;

Poor-Man's Profiler

joshpoley — Wed, 12 Mar 2008 18:59:00 GMT

Microsoft's C/C++ compiler supports the /Gh and /GH switches, these options allow the developer to inject a function call into the beginning and exit of every procedure being compiled. Aside from enabling some cool logging/traceability scenarios, you can also utilize this functionality to build in a simple profiler.

There have been a (surprisingly large) handful of times when I haven't had a profiler available, either due to the fact that one simply didn't exist for the platform I was working on, or a licensed copy of one wasn't readily obtainable. As such, having a simple profiler library ready to link into a project has been a welcome and useful addition to my toolbox.

Here is the basic outline of what needs to happen to pull this together. Also note that the discussion here assumes a 32bit Intel architecture. If you are on a 64bit or a non-Intel compatible platform, you will have to forgo the inline assembly, and supply the _penter/_pexit functions in a stand-alone assembly module.

1. Provide a _penter function

Per the example referenced in the /Gh documentation, you will need to link in your own _penter function which utilizes the naked calling convention. Because of this (and not shown in the official example), you will need to adjust the stack pointer to account for any local variables you use. For example, assuming we use 24 bytes of local stack:

// adjust our stack pointer for local variables

mov ebp, esp

sub esp, 24

In this function you will want to grab the return address off the stack (which will be used to identify the caller) and take a snapshot of the time. To store these values, you will need to use a stack-like data structure so you can keep track of the start times for all the functions in the current call chain. Have each thread utilize its own storage buffers for the timing stats, the last thing you want to do is introduce a deadlock with your profiling code (or corrupt memory if you don't protect yourself).

To get the time, use a high resolution clock. I would recommend either calling QueryPerformanceCounter() or invoke rdtsc from assembly if you can account for/mitigate the issues around using it directly.

2. Provide a _pexit function

In your custom _pexit function, grab the current time so you can calculate the run-time. Since file IO is very slow, write out the caller's address and the elapsed time to a separate buffer in memory.

Warning: If your _penter or _pexit functions call any other functions or methods (such as the above mentioned QueryPerformanceCounter), make sure those are not compiled with /Gh or /GH. Otherwise death by recursion will ensue.

Depending on how much run-time overhead you want to deal with, you can also add additional code to keep track of:

· hit counts (how many times a function is called)

· minimum and maximum times spent in a function

· the amount of time spent in calls to children

3. Write an "at end" function

Since our timings are still in memory, we need to provide a function which will be called at the end of the application to write the results out to a file.

4. Post-Processing

Write a tool to translate the caller addresses into the actual function name. This can either be a separate application or done within step 3 above. To do the translation, you can either utilize a map file or use the PDB APIs. Then it is just a matter of providing the results in an easy to view/sort manner. Below is a screen shot of some results taken from an application which searches through the metadata in photos (times shown are in clock ticks to avoid rounding or truncation issues, and in this specific sample, there happens to be 2175.21 ticks in a microsecond).

Testing Your PDB Files

joshpoley — Thu, 10 Jan 2008 20:52:00 GMT

Very little attention is often applied to PDB files, but if you do ship them to customers, then you will want to ensure they don't contain more information than is necessary. The debugging information stored here can be used to reverse engineer or attack your code, so it is worthwhile to take a precautionary step and verify (i.e. test) that there is no private data being exposed via your public symbol files.

You can use the Debug Interface functions to crack open and query your .pdb files to see exactly what information is being provided. Then it is just a matter of creating a simple tool which can be inserted into your release process to validate the files. As a straightforward sample, we will show how to use the DIA APIs to query for keywords which could be used by an attacker.

1 void KeywordList(IDiaSymbol *globalSym, LPOLESTR search)

2 {

3 IDiaEnumSymbols *searchEnum = NULL;

4 globalSym->findChildren(SymTagNull, search, nsCaseInRegularExpression, &searchEnum);

5 if(searchEnum)

6 {

7 LONG numItems = 0;

8 searchEnum->get_Count(&numItems);

10 for(LONG i=0; i<numItems; i++)

11 {

12 IDiaSymbol *sym = NULL;

13 searchEnum->Item(i, &sym);

15 if(sym)

16 {

17 BSTR symName = NULL;

18 sym->get_name(&symName);

20 printf(" %ws\n", symName);

22 SysFreeString(symName);

23 sym->Release();

24 sym = NULL;

25 }

26 }

28 searchEnum->Release();

29 searchEnum = NULL;

30 }

31 }

The "globalSym" pointer is the PDB files' main collection of symbols; to get this pointer, the typical call flow looks like:

· Create an IDiaDataSource object

· Call IDiaDataSource::loadDataFromPdb to open a specific PDB file

· Call IDiaDataSource::openSession

· Call IDiaSession::get_globalScope to get the global symbol container

On line 4 we do a case-insensitive wild-card search through the global scope (* and ? are the only supported wild-cards). If any matches are found, the result will be a collection stored in (and accessed by) the IDiaEnumSymbol interface.

The "Count" property (line 8) will tell us how many matches were found. From there we pull out each symbol using the "Item" method (line 13) and then query the symbol's name (line 18). Even though the documentation for get_name doesn’t mention anything about freeing up the returned BSTR, you will want to clean up the memory (line 22) using SysFreeString, otherwise you will end up with a memory leak.

We can then call this helper function, for example, to dump out all instances of the word "password", which is probably not something we would want to leave around for others to gather information about.

KeywordList(pGlobalSymbol, L"*password*");

Google’s Chart API

joshpoley — Mon, 07 Jan 2008 18:49:00 GMT

Throughout the software development cycle there is a wide array of areas where you need to generate various reports which include nifty looking charts. Whether you are generating performance reports, bug trends, tracking memory or CPU utilization, or following code coverage results; generating a meaningful graph is important for delivering and allowing the easy consumption of key pieces of information. Traditionally I would use the Excel COM objects to automate the generation of graphs from our data sets, but Google recently released their charting API (http://code.google.com/apis/chart/) so I figured I would take a look at writing a tool which would use Google’s interface to generate and save off the resulting image. Being on vacation provided a good opportunity to delve into this project, so here I will go into some snippets of code where I read in a comma delimited data file and generate the inputs to Google’s interface.

Standard Disclaimer: The code snippets below are for illustrative purposes and may not be "production" quality – this was a "quick and dirty" application for private usage.

Program Outline

This gives you an outline of what the main flow of my application looks like:

// parse through the command line options

// read in the entire text file

// count # of columns by scanning the first line for the delimiters

// count # of rows by counting all the new lines in the file

// create 2d array of doubles

// parse in the values, and keep track of min/max values

// build a google encoded value string for each row

// build the axis labels

// create the color string

// build the entire url string

// send http request

// save the returned file on 200 Ok

Notes and Interesting Snippets

To start with, we read the entire file into memory (as opposed to reading it in line by line), this makes it easier to handle cases where the input file is in Unicode, UTF-8, or ANSI. This also gives us the opportunity to do some pre-processing: we can determine how many rows and columns of data we have so we know the limits to our data set. Then we can allocate a 2D array up-front without having to worry about resizing.

Once we have our 2D array, we can parse through and read the data values

1 // parse in the values, and keep track of min/max values

2 char* index = dataStart;

3 unsigned row = 0;

4 unsigned col = 0;

5 char dataBreak[3] = {0};

6 dataBreak[0] = delim;

7 dataBreak[1] = '\n';

8 while(*index)

9 {

10 if(index >= fileBuffer + dataLen) break;

11 rawData[col][row] = atof(index);

12 maxData = max(rawData[col][row], maxData);

13 minData = min(rawData[col][row], minData);

15 size_t offset = strcspn(index, dataBreak);

16 if(offset == 0) break;

17 index += offset;

19 if(*index == '\0') break;

20 if(*index == '\n')

21 {

22 row++;

23 col = 0;

24 }

25 if(*index == delim) ++col;

26 ++index;

27 }

On line 6 we pull the data delimiter from a variable, this allows us to support files which are tab delimited (for example) instead of comma. Lines 12 and 13 keep track of the minimum and maximum value; this is needed as we will want to scale the data into the range of 0 to 61 for input into Google's API. Line 15 will scan past all the data until it hits a new line or the delimiter. If we were super cool we would want to skip over (if present) a column of text which contains x-axis information.

1 // build a google encoded value string for each row

2 dataRows.strcpy("s:");

3 for(unsigned c=0; c<numCol; c++)

4 {

5 for(unsigned r=0; r<numRows; r++)

6 {

7 dataRows.scatf("%c", GoogleEncode(rawData[c][r], minData, maxData));

8 }

9 if(c < numCol-1)

10 dataRows.strcat(",");

11 }

Here we loop through our data arrays and convert the raw data into the Google number scheme needed for a line chart. The "dataRows" variable is a string class, which allows concatenation of printf style formatting (using the scatf member function, as seen on line 7). So be sure to substitute your own favorite string class.

Here is the GoogleEncode helper function:

char GoogleEncode(double data, double minData, double maxData)

{

const char baseGoogle[] =

"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";

double range = maxData - minData;

if(range == 0.0)

return baseGoogle[0];

if(minData < 0.0) data += minData;

else data -= minData;

data = data * 61 / range;

return baseGoogle[(int)data];

}

If "range" is 0.0, then all the input values are the same so we can just return a constant value (and avoid the divide by zero). Otherwise we slide the data into the positive dimension, and then we scale them into a base 62 value and return the appropriate character. The only question I have for Google is why they put 0-9 at the end of their number system - I would definitely be interested to hear the story behind that decision.

Generating the y-axis labels is just a matter of pulling out some evenly spaced values between "minData" and "maxData”. A simple loop accomplishes this:

double range = maxData - minData;

for(double r=minData; r<=maxData; r+=range/min(10.0, numRows-1))

{

yAxis.scatf("|%1.1lf", r);

}

We cap the number of labels at 10 to prevent too many entries. You may want to scale this value up or down depending on the actual pixel height of the graph being generated. Also be sure to handle the case where minData == maxData.

Wrapping It Up

So given the following input file, my tool generated this line chart:

Normal, Linear

0, 0.5

0.01, 0.5

0.03, 0.5

0.1, 0.5

0.3, 0.5

0.7, 0.5

0.9, 0.5

0.97, 0.5

1, 0.5

0.97, 0.5

0.9, 0.5

0.7, 0.5

0.3, 0.5

0.1, 0.5

0.03, 0.5

0.01, 0.5

0, 0.5

http://chart.apis.google.com/chart?chs=300x200&cht=lc&chd=s:AABGSq27972qSGBAA,eeeeeeeeeeeeeeeee&chxt=y,x&chxl=0:|0.0|0.1|0.2|0.3|0.4|0.5|0.6|0.7|0.8|0.9|1.0|1:|1|2|3|4|5|6|7|8|9|10|11|12|13|14&chco=0000FF,770000

Given the flexibility of the Google API, there are plenty of options/settings which a tool like this could take into consideration, but we've definitely gone on long enough for a single post.