SGriffin's MAPI Internals : MFCMAPI

MFCMAPI and Old Versions of Visual Studio

Stephen Griffin — Thu, 09 Jul 2009 21:45:29 GMT

Ever since I went to the new Vista style icon for MFCMAPI, it’s not been possible to compile in Visual Studio 6. I’ve spent some time looking at this, and it appears the only clean way to get VS 6 working again is to remove compression from the icon, something I really don’t want to do. So I’m considering dropping support for Visual Studio 6 altogether. While I’m at it, it’s been a while since I tried to compile in VS 2003 or 2005, so I was thinking about dropping support for those two as well. The only compiler I’d test would be Visual Studio 2008 (and 2010 whenever it comes out). Note that other than the known break in VS 6 I’m not planning on actively breaking compilation for 2003 or 2005. I just wouldn’t test them, so over time, I probably would break them.

Before I do this – I wanted to get feedback. Would anybody care if MFCMAPI no longer compiled in Visual Studios 6, 2003, or 2005?

Exchange MAPI and the Form Manager

Stephen Griffin — Wed, 08 Jul 2009 15:27:13 GMT

Had a customer contact me recently asking why MAPIOpenFormMgr doesn’t work with the Exchange MAPI download. Actually, it wasn’t working in Exchange 2003 either. I’m not sure exactly when it last worked in Exchange’s implementation of MAPI. We might have to go all the way back to the Exchange Client to find a version of Exchange’s MAPI where it worked.

There was more to this story that I wanted to post. A few years ago, on a whim, I debugged why MAPIOpenFormMgr was broken. I found it was looking for a forms manager DLL that Exchange hadn’t shipped in a long long time. However, there was a mechanism for directing Exchange’s MAPI to use a different forms manager, and one of the DLLs Exchange shipped could act as a forms manager. Since this wasn’t likely supported, and no one was asking about it, I filed it away as an interesting tidbit.

Since I did get asked about it, my plan was to post the reg key that controls which forms manager is used with a warning that since it isn’t supported or tested, it’s quite possible it will break on you. However, when I went to try it out, I found that when you try it out with the Exchange MAPI download, you crash right away. This isn’t surprising considering no one’s run the code in 10+ years.

So there’s no point in posting the key, though someone who was really determined could figure it out pretty quickly, and with a little debugging even figure out which DLL to point at. I guess I’ll leave that as an exercise for the reader. The official answer here is that Exchange’s implementation of MAPI doesn’t include a functioning forms manager, so none of the forms management portions of MAPI, such as MAPIOpenFormMgr or MAPIOpenLocalFormContainer, are available.

So Long Mapi Editor…And Thanks For All The Fish

Stephen Griffin — Mon, 15 Jun 2009 13:44:36 GMT

I’ve touched on the history of MFCMAPI before. The rebranding of MFCMAPI as MAPI Editor and posting it on the Microsoft Download site back in 2006 served a purpose, one for which at the time there were few alternatives. However, since then, I have taken back control over publishing of the tool so I can get updates and fixes to my users faster (as opposed to “never”, which was the case with that download).

Since setting up the Codeplex site, one thing that’s been bugging me is that the “MAPI Editor” download was still fairly popular. No amount of evangelizing the new builds on the Codeplex site would prevent people from searching for MFCMAPI on Microsoft.com and coming up with what I view as an old and buggy build. So, last week, we “fixed the glitch”. If you visit the download page now, instead of MAPI Editor, you’ll find a link to Codeplex. And if you download the file that’s there (apparently – there has to be a file to have a page on the download site), it’s a PDF that also directs you to Codeplex.

The switch was flipped last Thursday, and my download stats already show a huge spike. So the redirection must be working! Hopefully now I’ll stop seeing problem reports on that old build.

(Less Than) Portable Search Folders

Stephen Griffin — Thu, 21 May 2009 20:47:13 GMT

Let’s have fun with the object model and search folders:

Outlook running a profile in online mode.
Create this macro:
Sub TestAdvancedSearchComplete()

    Dim sch As Outlook.Search

    set sch = Application.AdvancedSearch("Inbox", "urn:schemas-microsoft-com:office:office#Keywords like 'Test'", True, "MySearchFolder")

    sch.Save ("MySearchFolder")

End Sub
Run the macro, observe the folder.
On the same machine, create a new profile in cached mode and start Outlook
Run macro, get “Run-time error '-2147219964 (80040604)': Cannot create folder.”

If we want to have more fun, instead of running the macro the second time, we can locate the folder “MySearchFolder” under the Search Folder node in Outlook. If we click on it, Outlook creates a search folder, but it doesn’t populate with the items we expect to find.

What’s going on here? To answer that, we have to know a bit about how Outlook’s persisted search folder feature works. When you create a search folder, either using the Object Model, or directly in Outlook, the folder isn’t the first thing created. The first thing created is the search folder definition message. This is a message with the message class “IPM.Microsoft.WunderBar.SFInfo” which lives in the associated contents of the Common Views folder. We can use MFCMAPI to take a look at this message:

And we can use MFCMAPI’s Smart View feature to parse the search folder definition stored in PR_WB_SF_DEFINITION:

From this, we find that the search folder definition contains the restriction that will form the basis of the search folder:

Restriction:
lpRes->rt = RES_CONTENT
lpRes->res.resContent.ulFuzzyLevel = FL_IGNORECASE | FL_FULLSTRING = 0x00010000
lpRes->res.resContent.ulPropTag = 0x8010101F (PT_MV_UNICODE)
lpRes->res.resContent.lpProp->ulPropTag = 0x8010001F (PT_UNICODE)
lpRes->res.resContent.lpProp->Value = Test
  Alt: cb: 8 lpb: 5400650073007400

Note that this restriction includes a property in the 0x8000 range, meaning it’s a named property. In fact, it’s the named property “Keywords” in the PS_PUBLIC_STRINGS namespace. Note also that there’s nothing in the restriction which tells me this. I only know it because I created the search folder and I know what the search folder is supposed to be looking for.

Now – what happens when we go to a different profile looking at the same mailbox? The search folder itself doesn’t get synched back to the server. Only this search folder definition message gets synced. When Outlook encounters this message, it places a dummy node under Search Folders and waits for the user to click on it. If the user does click on it, it builds the search folder using the information from PR_WB_SF_DEFINITION.

And that’s where the problem comes in: Named property mappings are store specific. A named property mapping that’s valid for one store will most likely be invalid for another store. When we switched to cached mode, we’re now working with the OST, not Exchange. Even if there happens to be an 0x8010101F property in this store, it’s not going to be the “Keywords” property. So when we clicked on the folder in Outlook, it created a search folder that searched for a bogus property.

What happened when we ran the macro is even more interesting: Outlook looked at our search string and built a search folder definition message with the appropriate PR_WB_SF_DEFINITION. However, it couldn’t save the message because it already existed, with a different definition!

Why don’t we have a problem when we create search folders directly in Outlook? The Outlook user interface severely restricts the kinds of restrictions you can set up. Because of that, Outlook is able to store the PR_WB_SF_DEFINITION in a different format (using SFST_FILTERSTREAM instead of SFST_MRES). That format is more flexible with named properties, but isn’t capable of storing the wide variety of restrictions that could be specified through the Outlook Object Model.

Workaround

We did look at taking a fix for this, but the limitations of SFST_FILTERSTREAM eliminated it as a possibility for a fix. And the fact that this problem only happens if the restriction contains a named property (IE it won’t repro on a search for subject or recipients) limited the scope of the problem.

However, for the case where the search folder is being created by a macro and we’re getting the “Cannot create folder” error, we found a neat workaround. We can use Outlook’s PropertyAccessor to delete the search folder definition message so it can be recreated. Here’s a sketch of the code:

Sub DeleteSFItem()
    Dim CommonViewsEIDBin As String
    Dim CommonViewsEIDString As String
    Dim CommonViewsFolder As Folder
    Dim ACTable As Table
    Dim oRow As Row
    Dim SFDefinitionEID As String
    Dim SFDefinitionItem As StorageItem
        
    CommonViewsEID = Session.DefaultStore.PropertyAccessor.GetProperty( _
        "http://schemas.microsoft.com/mapi/proptag/0x35E60102")
    CommonViewsEIDString = Session.DefaultStore.PropertyAccessor.BinaryToString( _
        CommonViewsEID)
    Set CommonViewsFolder = Session.GetFolderFromID(CommonViewsEIDString)
    Set ACTable = CommonViewsFolder.GetTable( _
        "[Subject] = 'MySearchFolder'", olHiddenItems)
    
    Set oRow = ACTable.GetNextRow()
    If (Not (oRow Is Nothing)) Then
        SFDefinitionEID = oRow("EntryID")
        Set SFDefinitionItem = Session.GetItemFromID(SFDefinitionEID)
        SFDefinitionItem.Delete
    End If
End Sub

For further reading on the inner workings of search folder definition messages, check out the Exchange Protocol Doc [MS-OSOSRCH].

May 2009 Release of MFCMAPI

Stephen Griffin — Wed, 20 May 2009 14:26:08 GMT

The May 2009 Release (build 6.0.0.1013) is live: http://mfcmapi.codeplex.com.

The coolest new feature has to be the addition of Smart View parsing to the Hex Editor. I actually had the idea to do this the morning I published the March 2009 release, but didn’t want to delay that release any longer. One side effect of exposing the parsing in the Hex Editor is I was able to do much better testing of the existing parsing engine and fix a bunch of memory leaks and crashes.

Here's a change list - see the Issue Tracker on Codeplex for more details, or look at the code:

Added ability to copy attachments from one message to another
Added Outlook version feature - uses MSI to retrieve installed version of Outlook
Added Smart View to hex editor
Fixed issue converting MAPI to MIME from an MSG file
Fixed parsing of fuzzy level flags
Updated MAPI headers
Removed extraneous pointer address output from restriction parsing
Added smart view parsing for Entry Lists, Properties, Restrictions, and Search Folder Definitions
Cleaned up memory leaks and crashes in smart view parsing

Enjoy.

UST and Long File Names

Stephen Griffin — Fri, 20 Mar 2009 15:26:16 GMT

I think this will be the last article to come from James’ post on debugging MAPI (previous posts here, here, and here). This last one bit me when I was trying to gather a UST trace from a customer.

Here’s the deal: The image file name associated with an executable can be different depending on how the process is launched. And Image File Execution Options (IFEO) works on the image file name, not the executable name. To illustrate this, we can rename mfcmapi.exe (my favorite executable, natch) to ThisIsAReallyLongName.exe and run some tests. First, let’s determine the short file name for this file:

D:\MFCMAPI>dir /x ThisIsAReallyLongName.exe
03/16/2009  10:28 AM         1,982,976 THISIS~1.EXE ThisIsAReallyLongName.exe

Next, we launch Procmon and look for anyone opening an IFEO key for a process with “this” in the name. Let’s try launching the process from the command line, using both the long and the short name:

D:\MFCMAPI>ThisIsAReallyLongName.exe
64-bit cmd.exe RegOpenKey
  HKLM\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\ThisIsAReallyLongName.exe
D:\MFCMAPI>THISIS~1.EXE
64-bit cmd.exe RegOpenKey
  HKLM\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\ThisIsAReallyLongName.exe

I trimmed and formatted the Procmon output a bit for clarity. The columns are bitness, process name, operation, and key read. So far, no difference in the keys. Let’s see what happens if we ask the debugger to launch the process:

D:\MFCMAPI>windbg ThisIsAReallyLongName.exe
64-bit windbg.exe RegOpenKey
  HKLM\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\ThisIsAReallyLongName.exe
32-bit ThisIsAReallyLongName.exe RegOpenKey
  HKLM\SOFTWARE\Microsoft\WINDOWS NT\CURRENTVERSION\Image File Execution Options\ThisIsAReallyLongName.exe
D:\MFCMAPI>windbg THISIS~1.EXE
64-bit windbg.exe RegOpenKey
  HKLM\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\THISIS~1.EXE
32-bit THISIS~1.EXE RegOpenKey
  HKLM\SOFTWARE\Microsoft\WINDOWS NT\CURRENTVERSION\Image File Execution Options\THISIS~1.EXE

The first thing we notice is now both the launching process and the launched process are reading IFEO keys. Next, we notice the 64/32 bit difference showing up in the paths. But the biggest difference is one is looking for a key named “ThisIsAReallyLongName.exe” while the other looks for “THISIS~1.EXE”.

So maybe this is a quirk of the debugger? Let’s try running the app from Start Run:

Start\Run: d:\mfcmapi\ThisIsAReallyLongName.exe
64-bit Explorer.EXE RegOpenKey
  HKLM\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\ThisIsAReallyLongName.exe
Start\Run: d:\mfcmapi\thisis~1.exe
64-bit Explorer.EXE RegOpenKey
  HKLM\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\thisis~1.exe

We’re back to one process reading the keys, but we still see that the key name depends on whether we used the long or short name for the file.

So – the upshot here, combined with the 32/64 bit issue from before, is when setting IFEO options to enable UST (or anything else IFEO is used for), you potentially need to set the options in 4 places, depending on the bitness of the process, whether or not a long file name is involved, and who launched the process.

March 2009 Release of MFCMAPI

Stephen Griffin — Wed, 18 Mar 2009 16:48:41 GMT

The March 2009 Release (build 6.0.0.1012) is live: http://mfcmapi.codeplex.com. Note the new URL – the old one still works.

At the urging of Jason, I’ve added parsing of entry IDs per the protocol docs.

Here's a change list - see the Issue Tracker on Codeplex for more details, or look at the code:

Super ultra cool entry ID parsing
Fixed slow dialogs
Fixed overnight crashes
Added Fast Shutdown (note – MFCMAPI won’t call Fast Shutdown normally – only if you ask it to)
Added support for fiddling with PST paths and setting passwords

Enjoy.

UST and 64 Bit Machines

Stephen Griffin — Tue, 17 Mar 2009 14:07:00 GMT

Digging more into lessons learned from James’ blog on analyzing memory usage (my first two articles are here and here). Today we take a side trip into 64 vs. 32 bit. While we were working on this issue, I decided to get some UST enabled dumps of MFCMAPI so I could compare a known scenario to the customer’s unknown scenario (a great diagnostic technique). However, no matter what settings I specified in gflags, when I attached the debugger, MFCMAPI didn’t have UST enabled!

Let’s look at how gflags works. When you run the command “gflags /i mfcmapi.exe +ust”, some registry keys are created here:

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options

Specifically, the key mfcmapi.exe is created and some values are set under it. When you launch a process, the OS matches looks for this registry key, reads the values, and uses them in setting up the process.

Of course, this didn’t work for me, so I ran Procmon to see what key the OS was looking for. Procmon showed the OS was looking here:

HKLM\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\Image File Execution Options

The key difference of course being the Wow6432Node. It turns out I had both the 32 bit and 64 bit debugger packages installed on this machine, and each comes with it’s own version of gflags. On a 32 bit machine, the 32 bit gflags sets the normal, non-wow64 key. But on a 64 bit OS, the 32 bit version of gflags gets redirected to Wow6432Node. The 64 bit version of gflags, of course, sets the non-wow64 key.

The trick here is I had run the 64 bit gflags, which set the non-wow64 key, but the MFCMAPI process was looking under Wow6432Node. Once I ran the 32 bit version of gflags, UST worked quite nicely. Problem solved.

Except – when I launched MFCMAPI from the 64 bit debugger a strange thing happened. The OS read from the non-wow64 node. If I’ve only run the 32 bit gflags I don’t get UST. The explanation is spelled out in Mithun’s post on Image File Execution Options (IFEO)– where the OS looks depends on the bitness of the process that called CreateProcess, not the bitness of the process being launched!

So – if you’re giving someone instructions to gather dumps for you, and you want UST (or any other IFEO setting), to be safe you’re best off having them set reg keys in both places. Then it won’t matter how the process was launched, or what the bitness is.

Next: UST and long file names.

Crash Running MFCMAPI Overnight

Stephen Griffin — Wed, 11 Mar 2009 14:52:12 GMT

So – one of my teammates tells me that if he leaves MFCMAPI running overnight, it has usually crashed by the time he comes in the next morning. We got a few dumps, but they were inconclusive since the stacks were completely different every time. Since we didn’t know what was going on, we started paying attention to when and where the problem happened. We noted the following:

He usually runs MFCMAPI from a tools share on one of my machines.
If he copies MFCMAPI locally and runs it overnight, it never crashes.
If he tries running some other tools from the same share, he can cause them to crash overnight as well.
Another engineer who overheard us discussing the problem was also able to reproduce it.

Armed with this information, I decided to look at the dumps again. The exception thrown in each was 0xC0000006, STATUS_IN_PAGE_ERROR. Ok – so that’s explains things – maybe some network hiccup causes us to lose the connection to the original executable, then the next time the process needs to load a page of memory it can’t find it?

For a while, I thought this was the end of the investigation – there’s no way MFCMAPI can control network hiccups, right? Then I stumbled across this linker option:

/SWAPRUN

Basically, when this option is set, the OS will copy the whole binary image to a local swap file before running it. We set the “Swap Run From Network” switch on a test build, and he hasn’t been able to reproduce the crash again. If you’re in to manually editing your .vcproj file, this is the equivalent of adding the line

SwapRunFromNet=”true”

to the “VCLinkerTool” section of your project’s configuration.

The next version of MFCMAPI will have this option enabled for all configurations.

Kernel Debugging MFCMAPI

Stephen Griffin — Fri, 06 Mar 2009 15:19:32 GMT

I noticed recently that one on of my machines, whenever I ran MFCMAPI and displayed one of my CEditor dialogs, the whole machine froze for 1 to several minutes. I figured this would be a fun opportunity to practice my kernel debugging skills and share what I learned.

A bit about my setup: I keep two physical machines in my office. On the larger machine, I run Hyper-V Server 2008. This headless box sits in the corner and hosts all my VMs, including the VM where I run Outlook and other day-to-day case and labor tracking tools on Vista. The smaller machine, running Windows Server 2008, is my file server and my primary development and debugging desktop. Both machines are x64 and many of my VMs are as well.

The machine where I saw the hang was my tools VM, so I figured I’d try a remote VM kernel debug from my development machine. I haven’t seen a good set of instructions for doing this with these operating systems, so I had to crib bits and pieces from various sources. The first step is to enable kernel debugging on the target machine. On Vista, this means using BcdEdit:

Open an administrative command prompt.
Run bcdedit /debug on. This enables kernel debugging using the default setting of COM1, which is what we need.

Next, we have to map a named pipe on the Hyper-V server to COM 1 on the VM. This named pipe is what we’ll connect to. The process is described here:

Open Hyper-V Manager, select the VM to be debugged, and open Settings
Select COM 1, choose Named pipe, and enter a name for the pipe. The name doesn’t really matter. I chose debug.

Now’s a good time to reboot the VM so both settings take effect.

If we try to skip ahead to the remote debug at this point, we won’t be able to connect. The reason is the firewall on the Hyper-V Server is going to block our attempts to hook up to the named pipe. Named pipes use the file server ports 139 and 445, so we just need to open those ports:

This is tricky since Hyper-V server is command line only. We can open a Remote Desktop to the server, or hookup a keyboard and monitor and log on, but either way all we’ll have is the command line.

Run the following two commands to open ports 139 and 445:

netsh advfirewall firewall add rule name="File Server (TCP 139)" localport=139 dir=in action=allow protocol=tcp

netsh advfirewall firewall add rule name="File Server (TCP 445)" localport=445 dir=in action=allow protocol=tcp

Now we’re ready to remote debug.

The following command, run from our remote machine, attaches the debugger (hyperboo is the name of my Hyper-V server, and Debug is the named pipe we attached to COM 1 on the target VM):

windbg -k com:pipe,port=\\hyperboo\pipe\Debug,resets=0,reconnect

Note that the debugger doesn't break in right away - it just sits there saying "Waiting to reconnect...". We run through the steps to hang the machine (start MFCMAPI, open Other/Options will do), then switch over to the debugger and hit Ctrl+Break to attach. Now the fun part begins. We point at the public NT symbols, then append the path where MFCMAPI's symbols are located:

.sympath srv*c:\symbols*http://msdl.microsoft.com/download/symbols
.sympath+ c:\mfcmapi\Release
.reload

We run !locks to see what's holding up the system. This command runs for a very long time as it scans memory for every lock:

1: kd> !locks
**** DUMP OF ALL RESOURCE OBJECTS ****
KD: Scanning for held locks...
Resource @ 0xfffffa80024ac0d0    Shared 1 owning threads
     Threads: fffffa80018cb290-01<*> 
KD: Scanning for held locks....................... 

Resource @ 0xfffffa8002423ab0    Exclusively owned
    Contention Count = 3017959
    NumberOfExclusiveWaiters = 8
     Threads: fffffa800255a3c0-01<*> 
     Threads Waiting On Exclusive Access:
              fffffa8003e57bb0       fffffa8001f326b0       fffffa800399fbb0       fffffa800356a8c0       
              fffffa8003ab5520       fffffa8003f1cbb0       fffffa800348abb0       fffffa80033a9060       

KD: Scanning for held locks. 

Resource @ 0xfffffa800324d8c0    Exclusively owned
    Contention Count = 538
     Threads: fffffa8003dd69f0-01<*> 
KD: Scanning for held locks. 

Resource @ 0xfffffa800250f3b0    Exclusively owned
    Contention Count = 266255
    NumberOfExclusiveWaiters = 2
     Threads: fffffa8003dd69f0-01<*> 
     Threads Waiting On Exclusive Access:
              fffffa800255a3c0       fffffa80032c0390       

KD: Scanning for held 

locks...
13422 total locks, 4 locks currently held

We find that lock 0xfffffa800324d8c0 has a contention count of 538, meaning it is blocking 538 threads! We then use the -v option to get more information on the lock:

1: kd> !locks -v 0xfffffa800324d8c0 

Resource @ 0xfffffa800324d8c0    Exclusively owned
    Contention Count = 538
     Threads: fffffa8003dd69f0-01<*> 

     THREAD fffffa8003dd69f0  Cid 134c.11ac  Teb: 000000007efdb000
        Win32Thread: fffff900c2dfa010 RUNNING on processor 0
     Not impersonating
     DeviceMap                 fffff8800ef7d720
     Owning Process            fffffa8003afaab0       Image:         MFCMapi.exe
     Attached Process          N/A            Image:         N/A
     Wait Start TickCount      6762745        Ticks: 28 (0:00:00:00.437)
     Context Switch Count      758                 LargeStack
     UserTime                  00:00:00.828
     KernelTime                00:00:04.281
     Win32 Start Address 0x0000000000eb318b
     Stack Init fffffa60035ecdb0 Current fffffa60035ebd30
     Base fffffa60035ed000 Limit fffffa60035e2000 Call 0
     Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
     Child-SP          RetAddr           Call Site
     fffffa60`035ec0c8 fffff960`000c5ba8 win32k!vAlphaPerPixelOnly+0x36
     fffffa60`035ec0d0 fffff960`000c57a3 win32k!AlphaScanLineBlend+0x304
     fffffa60`035ec190 fffff960`00259509 win32k!EngAlphaBlend+0x4f3
     fffffa60`035ec440 fffff960`00259a2a win32k!NtGdiUpdateTransform+0x1075
     fffffa60`035ec4f0 fffff960`00259b47 win32k!NtGdiUpdateTransform+0x1596
     fffffa60`035ec610 fffff960`00259d05 win32k!EngNineGrid+0xb3
     fffffa60`035ec6b0 fffff960`0025a17c win32k!EngDrawStream+0x1a1
     fffffa60`035ec770 fffff960`0025ab35 win32k!NtGdiDrawStreamInternal+0x440
     fffffa60`035ec820 fffff960`002737fc win32k!GreDrawStream+0x98d
     fffffa60`035ecac0 fffff800`0189cdf3 win32k!NtGdiDrawStream+0x9c
     fffffa60`035ecc20 00000000`75b8a59a nt!KiSystemServiceCopyEnd+0x13

1 total locks, 1 locks currently held

Note that MFCMAPI.exe holds this lock, so we're on the right track. We use .thread to switch our context to thread fffffa8003dd69f0, using the /P and /r switches to force the debugger to translate page table entries and reload symbols for the implicit process:

1: kd> .thread /P /r fffffa8003dd69f0
Implicit thread is now fffffa80`03dd69f0
Implicit process is now fffffa80`03afaab0
.cache forcedecodeptes done
Loading User Symbols
.....
Loading Wow64 Symbols
......................................

Now - we're stuck. Normally, we'd expect to see the user mode stack below the switch to kernel mode. But we don't see it here. Instead, we see the wow64 translation layer, since we're running a 32 bit process on a 64 bit processor:

1: kd> kL
  *** Stack trace for last set context - .thread/.cxr resets it
Child-SP          RetAddr           Call Site
fffffa60`035ec0c8 fffff960`000c5ba8 win32k!vAlphaPerPixelOnly+0x36
fffffa60`035ec0d0 fffff960`000c57a3 win32k!AlphaScanLineBlend+0x304
fffffa60`035ec190 fffff960`00259509 win32k!EngAlphaBlend+0x4f3
fffffa60`035ec440 fffff960`00259a2a win32k!NtGdiUpdateTransform+0x1075
fffffa60`035ec4f0 fffff960`00259b47 win32k!NtGdiUpdateTransform+0x1596
fffffa60`035ec610 fffff960`00259d05 win32k!EngNineGrid+0xb3
fffffa60`035ec6b0 fffff960`0025a17c win32k!EngDrawStream+0x1a1
fffffa60`035ec770 fffff960`0025ab35 win32k!NtGdiDrawStreamInternal+0x440
fffffa60`035ec820 fffff960`002737fc win32k!GreDrawStream+0x98d
fffffa60`035ecac0 fffff800`0189cdf3 win32k!NtGdiDrawStream+0x9c
fffffa60`035ecac0 fffff800`0189cdf3 win32k!NtGdiDrawStream+0x9c
fffffa60`035ecc20 00000000`75b8a59a nt!KiSystemServiceCopyEnd+0x13
00000000`001ddda8 00000000`75d7a996 wow64win!NtGdiDrawStream+0xa
00000000`001dddb0 00000000`75d63688 wow64!Wow64SystemServiceEx+0xca
00000000`001de660 00000000`75d7ab46 wow64cpu!ServiceNoTurbo+0x28
00000000`001de6f0 00000000`75d7a14c wow64!RunCpuSimulation+0xa
00000000`001de720 00000000`77ac52d3 wow64!Wow64LdrpInitialize+0x4b4
00000000`001dec80 00000000`77ac5363 ntdll!LdrpInitializeProcess+0x14ac
00000000`001def30 00000000`77ab85ce ntdll! ?? ::FNODOBFM::`string'+0x1ff19
00000000`001defe0 00000000`00000000 ntdll!LdrInitializeThunk+0xe

So we try .thread again, this time passing the /w switch to change to the 32 bit context:

1: kd> .thread /P /r /w fffffa8003dd69f0
Implicit thread is now fffffa80`03dd69f0
Implicit process is now fffffa80`03afaab0
.cache forcedecodeptes done
Loading User Symbols
.....
Loading Wow64 Symbols
......................................
The context is partially valid. Only x86 user-mode context is available.
x86 context set
0: kd:x86> kbL 1d
ChildEBP          RetAddr           Args to Child                                         
0040e05c 7732891e e8011177 00000060 0040e1e4 GDI32!NtGdiDrawStream+0x15
0040e160 73a6276e e8011177 00000060 0040e1e4 GDI32!GdiDrawStream+0x436
0040e3dc 73a624f9 00486ff8 004856c8 00000001 uxtheme!CImageFile::DrawBackgroundDS+0x439
0040e458 73a623a0 00486ff8 02900968 e8011177 uxtheme!CImageFile::DrawImageInfo+0x1dd
0040e4a8 73a627f8 02900968 e8011177 00000002 uxtheme!CImageFile::DrawBackground+0x45
0040e4f8 73a64946 02900968 e8011177 00000001 uxtheme!DrawThemeBackground+0xcc
0040e590 73a6789b 34011934 0292725c e8011177 uxtheme!CThemeWnd::NcPaintCaption+0x11a
0040e5b0 73a62ea0 0292725c 00000000 00000000 uxtheme!OnDwpSetIcon+0x6b
0040e604 73a60e74 00000000 00000000 00000001 uxtheme!_ThemeDefWindowProc+0x15d
0040e620 777dc18b 001d02de 00000080 00000001 uxtheme!ThemeDefWindowProcA+0x18
0040e668 777f7fec 001d02de 00000080 00000001 USER32!DefWindowProcA+0x68
0040e680 777e6050 0110d2f0 00000000 00000080 USER32!DefWindowProcWorker+0x27
0040e6c8 777f7fe3 0110d2f0 00000000 00000080 USER32!DefDlgProcWorker+0x7f7
0040e6e8 777d8807 001d02de 00000080 00000001 USER32!DefDlgProcA+0x29
0040e714 777d8962 77ca3db0 001d02de 00000080 USER32!InternalCallWinProc+0x23
0040e78c 777dc62c 00000000 77ca3db0 001d02de USER32!UserCallWinProcCheckWow+0x109
0040e7c4 777dd163 77ca3db0 001d02de 00000080 USER32!CallWindowProcAorW+0xa9
0040e7e4 00e75ed8 77ca3db0 001d02de 00000080 USER32!CallWindowProcA+0x1b
0040e804 00e76074 00000080 00000001 1ee70f3d MFCMapi!CWnd::DefWindowProcA+0x44
0040e820 00e08cb2 00000080 00000001 1ee70f3d MFCMapi!CWnd::WindowProc+0x3b
0040e850 00e79210 00000080 00000001 1ee70f3d MFCMapi!CEditor::WindowProc+0x222
0040e8b8 00e7929f 00000000 001d02de 00000080 MFCMapi!AfxCallWndProc+0x9a
0040e8d8 777d8807 001d02de 00000080 00000001 MFCMapi!AfxWndProc+0x36
0040e904 777d8962 00e79269 001d02de 00000080 USER32!InternalCallWinProc+0x23
0040e97c 777dc4b6 00000000 00e79269 001d02de USER32!UserCallWinProcCheckWow+0x109
0040e9c0 777deae2 0110d2f0 00000000 00e79269 USER32!SendMessageWorker+0x55b
0040e9e4 00e08f87 001d02de 00000080 00000001 USER32!SendMessageA+0x7f
0040edbc 00e74666 0040edf0 777d8807 001d02de MFCMapi!CEditor::OnInitDialog+0x167
0040edc4 777d8807 001d02de 00000110 001d02de MFCMapi!AfxDlgProc+0x31

Now we can piece together what happened. Down in the kernel layer, we see we’re doing something with graphics.Scanning back up the user mode stack, we find that MFCMapi!CEditor::OnInitDialog has called SendMessage, which is then handled by uxtheme. The message we're passing is 0x80, which is WM_SETICON. Back in the MFCMAPI code, we see we're on the line:

	SetIcon(m_hIcon, TRUE);			// Set big icon

Now it's all starting to come together. In the process of setting up out dialog, we call WM_SETICON twice, once to set the big icon and once to set the small icon. But this is a dialog, so the big icon isn't needed at all. When we remove this call, and just leave the call to set the small icon, dialogs open rapidly! This fix will be in the next release of MFCMAPI.

We learned three things in this process. First – setting up a live kernel debug to a Hyper-V VM isn’t all that hard once you know how to do it. Second, it is possible for ordinary user mode operations to hang in kernel mode. And last, copying and pasting code around without understanding what it does is never a good idea.

Reading an RTF Stream

Stephen Griffin — Wed, 04 Mar 2009 15:17:18 GMT

Suppose you’ve gotten a stream from WrapCompressedRTFStream and want to read what’s in it. Your code might look something like this:

#define MAXBYTES 256
void ProcessStream(LPSTREAM lpStream)
{
   HRESULT hRes = S_OK;
   BYTE bBuf[MAXBYTES];
   ULONG ulNumBytes = 0;
   LARGE_INTEGER li = {0};
   hRes = lpStream->Seek(
      li,
      STREAM_SEEK_SET,
      NULL);
   if (S_OK == hRes) do
   {
      hRes = lpStream->Read(
         bBuf,
         MAXBYTES,
         &ulNumBytes);
      if (ulNumBytes > 0)
      {
         // do something with the bytes read
      }
   }
   while (ulNumBytes == MAXBYTES);
}

Note that in the do/while loop, the exit condition is when ulNumBytes is not equal to MAXBYTES, the number of bytes requested. This is consistent with the documentation for ISequentialStream::Read, which states:

“If the number of bytes returned is less than the number of bytes requested, it usually means the Read method attempted to read past the end of the stream.”

However, is this a best practice? When I took a look at our implementation of Read, I found the following comment:

“pcbRead < cb is not an error and does not necessarily mean the end of the file has been reached.”

If this is case, the your exit condition should be when ulNumBytes is zero. I’ve never seen WrapCompressedRTFStream behave this way in practice (and in fact, the code above was cribbed from MFCMAPI), but I have had a customer report it.

I did observe this behavior from a different stream though. In the old EDK, there was a function, HrTextFromCompressedRTFStreamEx, which works similar to WrapCompressedRTFStream. Instead of producing an unwrapped RTF stream, it goes one step further and converts the RTF to plain text. Normally, for EDK functions you can look at the source and see what’s going on, but this function is one of the few in the EDK that does not include source. The only way to call this function is to include the library rtflib32.lib, which shipped only in the EDK. So I’ll just have to tell you how the function works.

You might think this function works by calling WrapCompressedRTFStream, then parsing the resulting RTF, but it does not. Instead, it actually does all of the unwrapping and parsing in one pass over the source stream. The problem here is the parsing is somewhat rudimentary. There are a number of conditions that it can hit within the RTF to cause it to say “I’ve had enough – try again later”. It’ll output whatever it’s read so far, usually fewer bytes than were requested, without any indication of an error. If you call back in to it, it’ll give you more. One of the conditions I know of to cause this is if HrTextFromCompressedRTFStreamEx thinks it has detected a change in the code page.

So – if you’re working with streams in MAPI, the best practice here is to read until you can’t read any more. Here’s the corrected code (MFCMAPI will be fixed in the next update):

#define MAXBYTES 256
void ProcessStream(LPSTREAM lpStream)
{
   HRESULT hRes = S_OK;
   BYTE bBuf[MAXBYTES];
   ULONG ulNumBytes = 0;
   LARGE_INTEGER li = {0};
   hRes = lpStream->Seek(
      li,
      STREAM_SEEK_SET,
      NULL);
   if (S_OK == hRes) do
   {
      hRes = lpStream->Read(
         bBuf,
         MAXBYTES,
         &ulNumBytes);
      if (ulNumBytes > 0)
      {
         // do something with the bytes read
      }
   }
   while (ulNumBytes > 0);
}

Ignoring Notifications

Stephen Griffin — Wed, 18 Feb 2009 15:06:10 GMT

Suppose you want to register for notifications on all the visible folders in a user’s mailbox. One option would be to walk the list of folders in the hierarchy tree and register for notifications. That would work, but would be very inefficient. It also wouldn’t be dynamic, since you wouldn’t catch new folders as they’re created. And if you register for too many notifications, you might run into this problem. A more elegant solution is to register for notifications at the message store level. This is what one customer did, but recently they ran into a problem with one of the new features in Outlook 2007.

Before Outlook 2007, if someone gave you permissions to see their Calendar, when you opened their Calendar Outlook would have to make a direct connection to their mailbox server, even if your profile was running in Cached mode. This was never an ideal experience for the user, as they would have to wait while Exchange built the views for the folder. And if too many people shared the same calendar, you could run into a scenario where each time a user looked at the folder, some other user’s views were deleted to make room, essentially meaning access for everyone, including the owner of the mailbox, was slow. This is a nasty scenario that isn’t fun to troubleshoot or resolve. Cached mode lessens the problem for the owner of the mailbox; their access will always be fast. But it doesn’t help the other users.

With Outlook 2007, we added the idea of caching these shared folders. You may have seen the option in your mailbox settings:

This allows you to cache a copy of the other users Calendar, Contacts, Tasks and Notes folders in your own mailbox so you don’t have to connect to the Exchange server every time you want to look at them. The folders get updated periodically through the same incremental change synchronization (ISC) mechanism used to sync your own mailbox. (BTW – if you want to cache mail folders too, there’s a reg key for that.)

Since we’re caching these folders locally, the data has to live somewhere right? And where else but right there in your OST! Taking a look at my cached mode profile using MFCMAPI, we can see I have Jeff Garman’s Calendar cached:

This is where the customer ran into a problem. Remember they were registering for notifications store wide. In cached mode, that would include these folders under Shared Data, which they weren’t particularly interested in. Their question was how to identify which notifications were for these Shared Data folders so they could ignore them. Now, it turns out we haven’t documented many MAPI properties in this area, but there’s a way to approach this that doesn’t require any special knowledge. The key is recognizing that all the folders they do wish to monitor live under IPM_SUBTREE, and that folder is well documented. Here’s the approach we worked out:

Maintain a cache of known folder entry IDs. Each entry would be accompanied by a Boolean indicating whether the folder is under IPM_SUBTREE or not.
Seed the cache with the entry IDs for IPM_SUBTREE, marked as being under IPM_SUBTREE, and the root folder, marked as being not under IPM_SUBTREE.
Implement a function, IsUnderIPMSubtree, which checks if the target entry ID is in the cache, and returns the value it finds. Comparisons should be done with CompareEntryIDs.
If the entry ID is not in the cache, IsUnderIPMSubtree obtains the entry ID of the parent folder, and calls IsUnderIPMSubtree with that parent entry ID
The result of this recursive IsUnderIPMSubtree call is cached with the target entry ID, and the value is then returned.
Whenever a notification comes in, use the function IsUnderIPMSubtree to determine if the notification was interesting or not.

This algorithm makes the most sense to run in Cached mode, where CompareEntryIDs and OpenEntry will both be relatively cheap. Expected recursion level should be rather low, and the total number of folders opened over the lifetime of the algorithm is capped by the total number of folders in the store. In addition to filtering out notifications from the Shared Data folders, you’ll also eliminate any spurious notifications from any of the other non-visible folders.

Detecting a Password Protected PST

Stephen Griffin — Tue, 17 Feb 2009 18:01:50 GMT

If you’ve been working with PSTs for a while, you’re probably familiar with the properties PR_PST_PW_SZ_OLD and PR_PST_PW_SZ_NEW, given in mspst.h, which are used to access password protected PSTs and also to change the password. I had a customer recently who needed to know if a PST they had was password protected in the first place, so they could process them appropriately.

We did some research and found that if you added such a PST to a profile, then called OpenMsgStore on it with the MDB_NO_DIALOG flag set, you would get MAPI_E_FAILONEPROVIDER, which is pretty much the most generic error you can get in MAPI. Since we knew there were other reasons you could get this error, the question was whether we could distinguish this particular case. Most interfaces in MAPI support the function GetLastError. I set up a bit of code to try calling OpenMsgStore on a password protected PST, and took a look at the result of GetLastError. Here’s what I saw in the MAPIERROR structure:

ulVersion = 0x00000000
lpszError = 0x00000000
lpszComponent = Personal Folders
ulLowLevelError = 0x00000000
ulContext = 0x30060401

Note the value for ulContext. It turns out it’s a unique constant the PST provider will set when the PST was password protected and we could not display a dialog to prompt for credentials. Development has given permission to document this context, as well as one other:

MAPI Error Code	ulContext	Meaning
MAPI_E_FAILONEPROVIDER	0x30060401	The password from PR_PST_PW_SZ_OLD failed with Access Denied, and MAPI_NO_DIALOG was passed.
MAPI_E_LOGON_FAILED	0x30060402	The credentials attempted failed for some other reason.

If the credentials attempted failed with Access Denied and MAPI_NO_DIALOG wasn’t passed, the PST provider will attempt to prompt for credentials. The other context (0x30060402) should be rare, only occurring in out of memory or other unusual circumstances.

It turns out the PST provider is very good about setting errors with error contexts. Usually the top level MAPI error is sufficient to explain why a call failed, but if there are other ambiguous scenarios involving the PST provider, check the ulContext from GetLastError. If it’s unique, let me know the scenario and the context you got, and I’ll see if I can document it.

January 2009 Release of MFCMAPI

Stephen Griffin — Tue, 06 Jan 2009 15:59:29 GMT

The January 2009 Release (build 6.0.0.1011) is live: http://www.codeplex.com/MFCMAPI

Mostly bug fixes this time around. Most prominent was the inability to add a property to an item. Also, the Property Tag Editor needed some serious TLC.

Here's a change list - see the Issue Tracker on Codeplex for more details, or look at the code:

Augmented the internal named prop array to include the types for most standard named props. These types will show up in the Property Tag Editor when you type in the name of a named prop.
Added versioning to add-ins.
Fixed adding new properties.
Scrubbed and rewrote a good bit of the Property Tag Editor. “Lookup Named Prop” has been renamed to “Create Named Prop”. Named prop lookup is now automatic. If the named prop mapping exists it will be found. If it doesn’t exist, the “Create Named Prop” button will attempt to create it.

Enjoy.

Welcome to the Outlook 2007 MAPI Reference

Stephen Griffin — Tue, 02 Dec 2008 23:47:00 GMT

The Outlook 2007 MAPI Reference just went live on the MSDN. The docs are propagating their way through the servers, and should start showing up in ~~the table of contents and~~ searches soon. [update: propagation is complete - Live and Google should reindex soon]

This is the MAPI documentation refresh I've been talking about for the last few months. The short version of the story is that the Exchange team was ready to cut the MAPI documentation loose, but the Outlook team wanted to keep it around, so we had to transfer ownership from one team to another. Now, we could have just moved the docs from one node to another and been done with it, but instead we figured this would be a good time to scrub the docs and fix a lot of long standing technical, formatting and art issues.

Some highlights:

The What's New page: documents what's been added, changed, and removed from the docs. Notably, CMC and CDO documentation was cut, and nearly a thousand properties were added.
The Samples page: points to a number of updated MAPI samples, including MFCMAPI itself, which now has a go link: http://go.microsoft.com/fwlink/?LinkID=124154
The Properties node: Under this node you'll find a massive list of properties. We started with the original list from Exchange's MAPI documentation, and added just about every property we could think of. If a property is mentioned in the Exchange Protocol documentation, it's most likely documented here.
The Interfaces node: This is where you'll find all the MAPI interfaces, such as IMAPIProp. All interface documentation has been scrubbed, and many include references to where you can find the interface used in MFCMAPI.

My favorite addition to the MAPI documentation though, is the "Community Content" banner now on every page. Did we leave something out? Was there something we should have said about a particular property? Just add it to the bottom of the article. This is a great way to keep the content up to date, and we will be paying attention to the comments posted. If we get a chance to do another refresh down the road we may even incorporate the feedback.

Update: Angela is talking about this as well.

Enjoy!