Software Sleuthing : Automation

Automating Crash Dump Analysis: Some Final Thoughts

joshpoley — Mon, 23 Jun 2008 19:04:00 GMT

For a reference, here are some links to the previous parts in this series:

· Getting the Crash Details from a .DMP File

· Showing the Disassembly from a .DMP File

So now the question that begs to be answered is "how do we best leverage all this information?" Let's start by assuming we can stick all of it into a database for further (and easier) querying. This then opens up a whole slew of natural reports, such as, how often do we see similar types of crashes, or which functions or modules in our code base are the scariest.

If we add the ability to talk to a source control system, then we could start tying crashes to owners or developers based on the source control history (since we know the filenames and line numbers from the stack trace). This then gives us the ability to send out notification emails ("You have a new crash dump to look at") or to assign associated bugs to the correct individuals.

And speaking of bugs: If we are linked into the bug tracking system, we can then add support for some more automatic maintenance activities. For example, when a new crash dump comes in, we can look in our database and see if it is identical to something we've seen before. If it is, and any associated bugs are still active, we may just want to note the occurrence and not waste anybody's time by manually debugging the crash (we could automatically edit the bug to indicate that a new dump is available if needed). On the other hand, if the bug has already been "fixed", then we will want somebody to pay attention to the new crash dump.

Showing the Disassembly from a .DMP File (Automating Crash Dump Analysis Part 4)

joshpoley — Tue, 17 Jun 2008 23:31:00 GMT

When investigating a crash, it can often be beneficial to see the assembly leading up too, and even following the event. It turns out that generating the corresponding assembly at an address is pretty easy. The hard part is finding a good starting point for enumerating if we want to look backwards from a given address. On machine architectures that use a variable length instruction set, we can't necessarily walk backwards and expect to get the correct instruction listing, so we need to jump backwards to a know good location, then fast-forward to where we want to be.

This function will take the IDebugControl and IDebugSymbols interface pointers opened earlier, then dump out 'numLines' of assembly instructions preceding and following the crash at 'addr'. The input address can be taken from the stack frame or the exception record in the event information.

HRESULT DumpDisassembly(IDebugControl *control, IDebugSymbols *symbols,

ULONG64 addr, int numLines)

{

HRESULT hr = S_OK;

char asmBuffer[1024] = {0};

ULONG asmSize = 0;

ULONG64 endAddr = 0;

ULONG64 ip = addr;

ULONG64 funcStart = 0;

ULONG64 offset = 0;

ULONG procType = 0;

bool fixedInstructionLength = false;

printf("\nDisassembly\n");

// test for an invalid instruction pointer

if(addr == 0)

{

printf(" no data at address 0x00000000\n");

return hr;

}

// determine the algorithm to use

hr = control->GetEffectiveProcessorType(&procType);

if(SUCCEEDED(hr))

{

if(procType == IMAGE_FILE_MACHINE_ARM)

fixedInstructionLength = true;

// insert additional fixed length instruction

// platforms here

}

// locate the start of the function

symbols->GetNameByOffset(addr, NULL, 0, NULL, &offset);

funcStart = addr - offset;

// find the beginning address

if(fixedInstructionLength)

{

// if we are using a fixed length instruction set

// then we can easily step backward n lines

ip = addr;

hr = control->GetNearInstruction(addr, -numLines, &ip);

}

else

{

// on a variable length instruction architecture, we need

// to start at a known good location (funcStart in this case).

// Then walk each instruction until we get to the addr,

// keeping track of the history in a circular buffer.

ULONG64 *circBuffer = new(std::nothrow) ULONG64[numLines];

size_t index = 0;

if(circBuffer)

{

// initialize the buffer

for(int i=0; i<numLines; i++)

circBuffer[i] = 0;

// make sure there is at least one entry

circBuffer[0] = ip;

// walk from the start till the addr

ip = funcStart;

while(ip < addr)

{

circBuffer[index] = ip;

hr = control->GetNearInstruction(ip, 1, &ip);

if(FAILED(hr)) ++ip;

index = (index+1) % numLines;

}

// find the oldest (non-zero) entry

{

ip = circBuffer[index];

index = (index+1) % numLines;

} while(ip == 0);

delete[] circBuffer;

}

// make sure we don't go past the beginning of the funciton

if(ip < funcStart) ip = funcStart;

// dump out the instructions before and after the event

for(int lines=0; lines<numLines; )

{

hr = control->Disassemble(ip, DEBUG_DISASM_EFFECTIVE_ADDRESS,

asmBuffer, ARRAYSIZE(asmBuffer)-1, &asmSize, &endAddr);

if(FAILED(hr))

break;

if(ip == addr)

printf("> %s", asmBuffer);

else

printf(" %s", asmBuffer);

if(ip > addr) ++lines;

ip = endAddr;

}

return hr;

}

MSDN References

· GetEffectiveProcessorType

· GetNameByOffset

· GetNearInstruction

· Disassemble

Getting the Crash Details from a .DMP File (Automating Crash Dump Analysis Part 3)

joshpoley — Fri, 06 Jun 2008 20:48:00 GMT

Aside from a stack trace (the "where"), you probably want to know something about the crash (the "why" or the "how"). With the code below, we can query some of the basic details about what happened. From this we will at least be able to determine if the application attempted to write to a protected page, or if it decided to feel the effects of a division by zero.

HRESULT DumpEvent(IDebugControl *control, IDebugSymbols *symbols)

{

union ExtraInfo

{

DEBUG_LAST_EVENT_INFO_EXCEPTION exceptionInfo;

// as needed, we can add more of the

// DEBUG_LAST_EVENT_INFO_xyz structs here

};

HRESULT hr = S_OK;

ULONG type = 0;

ULONG procID = 0;

ULONG threadID = 0;

ExtraInfo extraInfo;

ULONG extraInfoUsed = 0;

char description[80] = {0};

ULONG bugCheckCode = 0;

ULONG64 bugCheckArgs[4] = {0};

ZeroMemory(&extraInfo, sizeof(extraInfo));

printf("\nFault Information\n");

// look for Bug Check data

hr = control->ReadBugCheckData(&bugCheckCode, &bugCheckArgs[0], &bugCheckArgs[1],

&bugCheckArgs[2], &bugCheckArgs[3]);

if(SUCCEEDED(hr))

{

printf(" Bug Check: %X (0x%08X, 0x%08X, 0x%08X, 0x%08X)\n",

bugCheckCode, bugCheckArgs[0], bugCheckArgs[1],

bugCheckArgs[2], bugCheckArgs[3]);

}

// get the fault information

hr = control->GetLastEventInformation(&type, &procID, &threadID,

&extraInfo, sizeof(extraInfo), &extraInfoUsed, description,

ARRAYSIZE(description)-1, NULL);

if(FAILED(hr))

goto cleanup;

printf(" Description: %s\n", description);

// if we hit an exception, and we understand the type of exception, write

// out some additional information

if((type == DEBUG_EVENT_EXCEPTION) &&

(extraInfoUsed >= sizeof(extraInfo.exceptionInfo)))

{

EXCEPTION_RECORD64 *er = &extraInfo.exceptionInfo.ExceptionRecord;

printf(" Type: %s\n", GetExceptionName(er->ExceptionCode));

if(er->ExceptionCode == EXCEPTION_ACCESS_VIOLATION ||

er->ExceptionCode == EXCEPTION_IN_PAGE_ERROR)

{

if(er->ExceptionInformation[0] == 0)

printf(" Read at: 0x%08X\n", er->ExceptionInformation[1]);

else if(er->ExceptionInformation[0] == 1)

printf(" Write at: 0x%08X\n", er->ExceptionInformation[1]);

else if(er->ExceptionInformation[0] == 8)

printf(" User Mode Fault: 0x%08X\n", er->ExceptionInformation[1]);

}

cleanup:

return hr;

}

The GetExceptionName function is a simple helper that you can provide, which just transforms the exception defines into human readable strings.

MSDN References

· ReadBugCheckData

· GetLastEventInformation

· EXCEPTION_RECORD

Getting the Stack from a .DMP File (Automating Crash Dump Analysis Part 2)

joshpoley — Mon, 02 Jun 2008 18:06:00 GMT

So now that we have a memory dump file, and know how to open it, we will want to pull some useful data out. To start with, we will grab the stack trace (which is arguably one of the more important pieces). And in case you've stumbled across the "dumpstk" sample in the WinDBG install\sdk directory, we will do things a little bit differently and not call OutputStackTrace() for these reasons:

1. By enumerating the stack structures directly, we have more control over how to format the results.

2. We don't have to deal with supplying an IDebugOutputCallbacks implementation (see the out.cpp file which accompanies the "dumpstk" sample).

3. We have individual control over each frame (OutputStackTrack can "chunk" up frames as they are sent to the IDebugOutputCallback interface).

Here is a sample function that takes pointers to the IDebugControl and IDebugSymbols interfaces (which are opened in part 1) and will dump the stack trace with some good old fashioned printf statements.

HRESULT DumpStack(IDebugControl *control, IDebugSymbols *symbols)

{

HRESULT hr = S_OK;

DEBUG_STACK_FRAME stackFrames[4096] = {0};

ULONG numFrames = 0;

// get the stack trace

hr = control->GetStackTrace(0, 0, 0, stackFrames, ARRAYSIZE(stackFrames),

&numFrames);

if(FAILED(hr))

goto cleanup;

printf("\nStack Trace\n");

// dump out the details for each function call

for(ULONG frame=0; frame<numFrames; frame++)

{

HRESULT symhr;

char name[512];

unsigned __int64 offset = 0;

ULONG lineNo = 0;

// get the module, function, and offset

ZeroMemory(name, ARRAYSIZE(name));

symhr = symbols->GetNameByOffset(stackFrames[frame].InstructionOffset,

name, ARRAYSIZE(name)-1, NULL, &offset);

if(SUCCEEDED(symhr))

printf("%s+0x%I64X", name, offset);

else

printf("0x%08I64X", stackFrames[frame].InstructionOffset);

// look for source file name and line number

ZeroMemory(name, ARRAYSIZE(name));

symhr = symbols->GetLineByOffset(stackFrames[frame].InstructionOffset,

&lineNo, name, ARRAYSIZE(name)-1, NULL, NULL);

if(SUCCEEDED(symhr))

printf("\n %s(%u)", name, lineNo);

printf("\n");

}

cleanup:

return hr;

}

Everything here is pretty straight forward: we retrieve the stack frame data, then just walk through each element of the populated array and pull out the module, the function name, and if we have full symbols, the filename and line number of the source code.

One thing you might want to add here is some intelligence about detecting recursive sequences. If a stack trace is extremely long and filled with repeating entries, it might be better to just write out the top and bottom of the stack and omit the majority of the redundant line items.

MSDN References

Opening a Crash Dump File (Automating Crash Dump Analysis Part 1)

joshpoley — Tue, 27 May 2008 19:20:00 GMT

So let's assume for the moment that you have a collection of crash dump files from your team's application. These files may be generated from your stress tests, betas, etc. But where they come from really isn't important, what is important is that we want some way to dig into these files and gather information, evaluate priority, etc., without having to manually open each one in the debugger.

Once you install WinDBG, you are given a COM interface allowing you to integrate debugging functionality into your own application. We can then use this to drive the task of opening and querying a crash dump file. When installing, just make sure you select the SDK option, which will give you the header files you will need. To make you life happier, you should also install the symbol packages for all the Operating Systems you support (also available from the above link) - doing so will give you function names from system binaries in the stack traces we will eventually create.

To start with we will go over the minimum code to initialize the debugging library and open a crash dump file.

#include <windows.h>

#include "dbgeng.h" // windbg header

int __cdecl main(int argc, char* argv[])

{

HRESULT hr = E_FAIL;

IDebugClient *client = NULL;

IDebugControl *control = NULL;

IDebugSymbols *symbols = NULL;

// Initialize COM

hr = CoInitialize(NULL);

if(FAILED(hr))

goto cleanup;

// Create the base IDebugClient object

hr = DebugCreate(__uuidof(IDebugClient), (LPVOID*)&client);

if(FAILED(hr))

goto cleanup;

// from the base, create the Control and Symbols objects

hr = client->QueryInterface(__uuidof(IDebugControl), (LPVOID*)&control);

if(FAILED(hr))

goto cleanup;

hr = client->QueryInterface(__uuidof(IDebugSymbols), (LPVOID*)&symbols);

if(FAILED(hr))

goto cleanup;

// we can supplement the _NT_SYMBOL_PATH environment variable

// by adding a path here

symbols->SetSymbolPath("c:\\myApp\\release");

// the debugger will need to look at the actual binaries

// so provide the path to the exsecutable files

symbols->SetImagePath("c:\\myApp\\release");

// open the crash dump

hr = client->OpenDumpFile("c:\\myApp\\myApp.dmp");

if(FAILED(hr))

goto cleanup;

// wait for the engine to finish processing

control->WaitForEvent(DEBUG_WAIT_DEFAULT, INFINITE);

cleanup:

// cleanup and destroy the objects

if(symbols)

{

symbols->Release();

symbols = NULL;

}

if(control)

{

control->Release();

control = NULL;

}

if(client)

{

client->Release();

client = NULL;

}

// cleanup COM

CoUninitialize();

return 0;

}

The semicolon separated path you pass into SetSymbolPath() should provide the location of the .pdb symbol files for all binaries consumed by your application. The same goes for the paths to the actual binary files themselves, which is passed in via SetImagePath(). Without the symbols or the actual binaries, your investigations won't be as successful.

MSDN References

· DebugCreate() which is used to instantiate the initial IDebugClient interface.

· IDebugClient

· IDebugControl

· IDebugSymbols

Prolific Usage of MiniDumpWriteDump (Automating Crash Dump Analysis Part 0)

joshpoley — Mon, 19 May 2008 18:27:00 GMT

As a precursor to a series on programmatically querying crash dump files, I wanted to first talk about the MiniDumpWriteDump function.

You don't necessarily need to wait for your application to crash in order to generate a memory dump. As soon as you detect something bad happening, you can call MiniDumpWriteDump yourself and generate a crash dump file on the fly. If your application likes to eat exceptions or ignore error codes, or important asserts fire, or special test cases fail, then this is a great place to start collecting failure data for offline analysis.

And to make it super easy for you, I've created a helper function which will generate a filename based on the computer's name along with the date and time, then invoke the crash dump generator.

#include <dbghelp.h>

HRESULT GenerateCrashDump(MINIDUMP_TYPE flags, EXCEPTION_POINTERS *seh=NULL)

{

HRESULT error = S_OK;

// get the time

SYSTEMTIME sysTime = {0};

GetSystemTime(&sysTime);

// get the computer name

char compName[MAX_COMPUTERNAME_LENGTH + 1] = {0};

DWORD compNameLen = ARRAYSIZE(compName);

GetComputerNameA(compName, &compNameLen);

// build the filename: APPNAME_COMPUTERNAME_DATE_TIME.DMP

char path[MAX_PATH] = {0};

sprintf_s(path, ARRAYSIZE(path),

"c:\\myapp_%s_%04u-%02u-%02u_%02u-%02u-%02u.dmp",

compName, sysTime.wYear, sysTime.wMonth, sysTime.wDay,

sysTime.wHour, sysTime.wMinute, sysTime.wSecond);

// open the file

HANDLE hFile = CreateFileA(path,

GENERIC_READ|GENERIC_WRITE,

FILE_SHARE_DELETE|FILE_SHARE_READ|FILE_SHARE_WRITE,

NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);

if(hFile == INVALID_HANDLE_VALUE)

{

error = GetLastError();

error = HRESULT_FROM_WIN32(error);

return error;

}

// get the process information

HANDLE hProc = GetCurrentProcess();

DWORD procID = GetProcessId(hProc);

// if we have SEH info, package it up

MINIDUMP_EXCEPTION_INFORMATION sehInfo = {0};

MINIDUMP_EXCEPTION_INFORMATION *sehPtr = NULL;

if(seh)

{

sehInfo.ThreadId = GetCurrentThreadId();

sehInfo.ExceptionPointers = seh;

sehInfo.ClientPointers = FALSE;

sehPtr = &sehInfo;

}

// generate the crash dump

BOOL result = MiniDumpWriteDump(hProc, procID, hFile,

flags, sehPtr, NULL, NULL);

if(!result)

{

error = (HRESULT)GetLastError(); // already an HRESULT

}

// close the file

CloseHandle(hFile);

return error;

}

Sample call:

GenerateCrashDump((MINIDUMP_TYPE)

(MiniDumpNormal |

MiniDumpWithHandleData |

MiniDumpWithUnloadedModules),

NULL);

If you happen to have OS exception information available (via __try/__except), then you will definitely want to pass in the EXCEPTION_POINTERS information. This will embed important and detailed information about the exception into the crash dump.

And if you want to get really fancy, you can embed additional (custom) information via the 2^nd to last parameter to MiniDumpWriteDump. For example, if your application is consuming external documents or information, it can be very handy to have this information (or metadata about it) accessible when diagnosing crash dumps after the fact.

Google’s Chart API

joshpoley — Mon, 07 Jan 2008 18:49:00 GMT

Throughout the software development cycle there is a wide array of areas where you need to generate various reports which include nifty looking charts. Whether you are generating performance reports, bug trends, tracking memory or CPU utilization, or following code coverage results; generating a meaningful graph is important for delivering and allowing the easy consumption of key pieces of information. Traditionally I would use the Excel COM objects to automate the generation of graphs from our data sets, but Google recently released their charting API (http://code.google.com/apis/chart/) so I figured I would take a look at writing a tool which would use Google’s interface to generate and save off the resulting image. Being on vacation provided a good opportunity to delve into this project, so here I will go into some snippets of code where I read in a comma delimited data file and generate the inputs to Google’s interface.

Standard Disclaimer: The code snippets below are for illustrative purposes and may not be "production" quality – this was a "quick and dirty" application for private usage.

Program Outline

This gives you an outline of what the main flow of my application looks like:

// parse through the command line options

// read in the entire text file

// count # of columns by scanning the first line for the delimiters

// count # of rows by counting all the new lines in the file

// create 2d array of doubles

// parse in the values, and keep track of min/max values

// build a google encoded value string for each row

// build the axis labels

// create the color string

// build the entire url string

// send http request

// save the returned file on 200 Ok

Notes and Interesting Snippets

To start with, we read the entire file into memory (as opposed to reading it in line by line), this makes it easier to handle cases where the input file is in Unicode, UTF-8, or ANSI. This also gives us the opportunity to do some pre-processing: we can determine how many rows and columns of data we have so we know the limits to our data set. Then we can allocate a 2D array up-front without having to worry about resizing.

Once we have our 2D array, we can parse through and read the data values

1 // parse in the values, and keep track of min/max values

2 char* index = dataStart;

3 unsigned row = 0;

4 unsigned col = 0;

5 char dataBreak[3] = {0};

6 dataBreak[0] = delim;

7 dataBreak[1] = '\n';

8 while(*index)

9 {

10 if(index >= fileBuffer + dataLen) break;

11 rawData[col][row] = atof(index);

12 maxData = max(rawData[col][row], maxData);

13 minData = min(rawData[col][row], minData);

15 size_t offset = strcspn(index, dataBreak);

16 if(offset == 0) break;

17 index += offset;

19 if(*index == '\0') break;

20 if(*index == '\n')

21 {

22 row++;

23 col = 0;

24 }

25 if(*index == delim) ++col;

26 ++index;

27 }

On line 6 we pull the data delimiter from a variable, this allows us to support files which are tab delimited (for example) instead of comma. Lines 12 and 13 keep track of the minimum and maximum value; this is needed as we will want to scale the data into the range of 0 to 61 for input into Google's API. Line 15 will scan past all the data until it hits a new line or the delimiter. If we were super cool we would want to skip over (if present) a column of text which contains x-axis information.

1 // build a google encoded value string for each row

2 dataRows.strcpy("s:");

3 for(unsigned c=0; c<numCol; c++)

4 {

5 for(unsigned r=0; r<numRows; r++)

6 {

7 dataRows.scatf("%c", GoogleEncode(rawData[c][r], minData, maxData));

8 }

9 if(c < numCol-1)

10 dataRows.strcat(",");

11 }

Here we loop through our data arrays and convert the raw data into the Google number scheme needed for a line chart. The "dataRows" variable is a string class, which allows concatenation of printf style formatting (using the scatf member function, as seen on line 7). So be sure to substitute your own favorite string class.

Here is the GoogleEncode helper function:

char GoogleEncode(double data, double minData, double maxData)

{

const char baseGoogle[] =

"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";

double range = maxData - minData;

if(range == 0.0)

return baseGoogle[0];

if(minData < 0.0) data += minData;

else data -= minData;

data = data * 61 / range;

return baseGoogle[(int)data];

}

If "range" is 0.0, then all the input values are the same so we can just return a constant value (and avoid the divide by zero). Otherwise we slide the data into the positive dimension, and then we scale them into a base 62 value and return the appropriate character. The only question I have for Google is why they put 0-9 at the end of their number system - I would definitely be interested to hear the story behind that decision.

Generating the y-axis labels is just a matter of pulling out some evenly spaced values between "minData" and "maxData”. A simple loop accomplishes this:

double range = maxData - minData;

for(double r=minData; r<=maxData; r+=range/min(10.0, numRows-1))

{

yAxis.scatf("|%1.1lf", r);

}

We cap the number of labels at 10 to prevent too many entries. You may want to scale this value up or down depending on the actual pixel height of the graph being generated. Also be sure to handle the case where minData == maxData.

Wrapping It Up

So given the following input file, my tool generated this line chart:

Normal, Linear

0, 0.5

0.01, 0.5

0.03, 0.5

0.1, 0.5

0.3, 0.5

0.7, 0.5

0.9, 0.5

0.97, 0.5

1, 0.5

0.97, 0.5

0.9, 0.5

0.7, 0.5

0.3, 0.5

0.1, 0.5

0.03, 0.5

0.01, 0.5

0, 0.5

http://chart.apis.google.com/chart?chs=300x200&cht=lc&chd=s:AABGSq27972qSGBAA,eeeeeeeeeeeeeeeee&chxt=y,x&chxl=0:|0.0|0.1|0.2|0.3|0.4|0.5|0.6|0.7|0.8|0.9|1.0|1:|1|2|3|4|5|6|7|8|9|10|11|12|13|14&chco=0000FF,770000

Given the flexibility of the Google API, there are plenty of options/settings which a tool like this could take into consideration, but we've definitely gone on long enough for a single post.

Creating Calendar Items

joshpoley — Mon, 10 Dec 2007 19:25:00 GMT

It is nice to be able to allow team mates to optionally add meeting requests or reminders to their calendars through an informational email or web page. For example, if you have a website showing information on how team members can run a stress tool on their PC at night, you may want to provide a link which will add a reminder to their calendar.

Most calendar applications support the iCalendar standard, defined by RFC 2445 (wiki), which are files with the ".ICS" extension. Using this file format you can easily (and even programmatically) create events which are added into associates’ calendars. Making it easy for people to help you out is important, as you will want to remove as many barriers as possible to improve the participation on things like stress, bug bashes, etc.

Calendar Entry

So let’s take a look at a simple “hello world” sample:

BEGIN:VCALENDAR

VERSION:2.0

PRODID:Josh Poley Blog Code

BEGIN:VEVENT

SUMMARY:Hello World

DTSTART;TZID=US-Pacific:20071210T090000

DTEND;TZID=US-Pacific:20071210T091500

END:VEVENT

END:VCALENDAR

iCalendar files are organized with a series of nested blocks which are denoted with the "BEGIN" and "END" tags. The block name (or what the block describes) is denoted by a keyword which will always start with the letter "V". All ICS files must contain a "VCALENDAR" block which is the main container.

VERSION:2.0

PRODID:Josh Poley Blog Code

Inside the VCALENDAR block, there must be a "VERSION" tag; we use 2.0 here as that references RFC 2445. Each VCALENDAR block must also include a "PRODID" tag, which is a string describing the generator of the iCalendar content.

A VEVENT block will describe an actual item on your calendar. The SUMMARY tag defines the subject of the entry and the DTSTART and DTEND tags specify when the event happens.

DTSTART;TZID=US-Pacific:20071210T090000

The date/time stamps are organized as such: An optional time zone string (which is denoted with the "TZID" string), followed by the date: YYYYMMDD. The letter "T" is used to separate the time which is in the form: HHMMSS. In this sample, we specify Pacific Time, and our calendar item starts at 9:00AM on December 10^th, 2007.

Reminders

Now, if we want to add a reminder, we will stick a VALARM block inside the VEVENT section:

BEGIN:VALARM

TRIGGER:-PT15M

ACTION:DISPLAY

END:VALARM

The "TRIGGER" tag defines when the reminder will go off, in our case we tell the calendar system to remind the user 15 minutes before the event. The "ACTION" tag denotes how the software should notify the user; here we tell the application to display a visual reminder.

Conclusion

Many testing automation tools make use of email to help coordinate the distribution of information. Adding calendar items into the mix may be appropriate for your organization and situation, and this post should give you enough to get started.

Disclaimer: The above samples work for me when using Outlook 2007. You may need to tweak them as necessary to get them working with your calendar software as various products implement the RFC with varying degrees of consistency.

Security Reminder: As with opening any attachments, use common sense and be sure you trust the source and the content before opening/executing files.

Edit 12/12/2007:

The next post shows how to setup IIS to serve up dynamic ICS files.