I know that it's been quite a while since I have written a blog. I have been plenty busy up here in the Pac-Northwest. Since my last post I have moved into new condo and back to the Server & Tools group to work in my original group - OneCare. Now that I've covered that, I give you one of the issues that I ran into a little while back.
During some standard processes I had some automation going out and running this binary that had been working fine for a while. A couple of days after the job kicked off I was on a server investigating something totally unrelated when I noticed that the exe that was being called (from a network share) was hanging. After checking a couple of other servers I found that it was hanging on them as well so I needed to investigate. I proceeded to investigate by attaching a debugger on the offending executable. You can see that the process had been up for over 2 days - but why? This had worked so well before.
1 Process Uptime: 2 days 6:09:16.000
Once I look at the stack backtrace I can see something immediately that is not right. The binary that I call from the network share is explicitly called without the GUI - so why are there windowing APIs on the call stack?
3 0:000> kbn
4
5 0 Id: 355ec.3488c Suspend: 1 Teb: 7ffdf000 Unfrozen
6 # ChildEBP RetAddr Args to Child
7 00 0012f758 7739d05f 77392c53 00000000 00000000 ntdll!KiFastSystemCallRet
8 01 0012f790 7738f152 4411008c 00000000 00000001 user32!NtUserWaitMessage+0xc
9 02 0012f7b8 773a16c2 77380000 00200838 00000000 user32!InternalDialogBox+0xd0
10 03 0012fa78 773a0fa4 0012fbe0 ffffffff ffff0000 user32!SoftModalMessageBox+0x94b
11 04 0012fbc8 7739df11 0012fbe0 00000000 7739de84 user32!MessageBoxWorker+0x2ba
12 05 0012fc34 735df73f 0012fc4c 00000000 00000000 user32!MessageBoxIndirectA+0xb8
13 06 0012fc74 735df5bb 0012fcb8 00000000 00000000 msvbvm60!AppCtlProc+0x2b
14 07 0012fc9c 735df8b6 735df6e4 0012fcb8 0012fcb8 msvbvm60!APP::GetPropA+0x115
15 08 0012fccc 735d3c78 00000000 00bc0528 0012fcf4 msvbvm60!_HrGetFileName+0x6f
16 09 0012fd30 735d3cc6 00bf1e94 00bb1fa4 00000030 msvbvm60!CreateEmbeddedObject+0x149
17 0a 0012fd50 7363c331 00168aa4 00000000 00000030 msvbvm60!CreateEmbeddedObject+0x197
18 0b 0012fd6c 7363c613 00168aa4 00000000 00000030 msvbvm60!EpiGetInvokeArgs+0x40f
19 0c 0012fed4 73577a5a 0014ee38 0040286c 00bb1e94 msvbvm60!CoerceArg+0x19b
20 0d 0012fef8 73573959 0040286c 00000000 0040286c msvbvm60!LIBID30_CDAO+0xa
21 0e 0012ff18 735736d2 00400000 00400000 00000000 msvbvm60!_CEDIT_vtbl::`vftable'+0x131
22 0f 0012ff38 735735d8 00000000 00400000 00000000 msvbvm60!_CDRIVE_vtbl::`vftable'+0x4a
23 *** ERROR: Module load completed but symbols could not be loaded for [CONFIDENTIAL]
24 10 0012ffb8 00402736 0040286c 77e523e5 00000000 msvbvm60!_CDIR_vtbl::`vftable'+0x100
25 WARNING: Stack unwind information not available. Following frames may be wrong.
26 11 0012fff0 00000000 0040272c 00000000 78746341 [CONFIDENTIAL]+0x2736
Next I dump the stack with the parameters and since I have all the symbols I can see that there is a structure that is of interest which I suspect will have the text of the window that is hanging there and cannot be seen by logging in.
1 0:000> kP
2 <snip>
3 0012fa78 773a0fa4 user32!SoftModalMessageBox(
4 struct _MSGBOXDATA * lpmb = 0x0012fbe0)+0x94b
5 </snip>
I run dt (display types) on the struct to dump our the contents so that I can get the address of the string that cantains the message text. (requires private symbols)
34 0:000> dt _MSGBOXDATA 0x0012fbe0
35 user32!_MSGBOXDATA
36 <snip>
37 +0x00c lpszText : 0x002007e0 -> 0x52
38 </snip>
Now that I have the address of the string I can dump that out with the SIE extention's !showstringw (double-byte words) command.
40 0:000> !showstringw 0x002007e0
41 Run-time error '70':Permission denied
What do have we here! A security problem? But wait didn't we already execute the application? We already have access to the filesystem so what's the problem? I look deeper in the stack at the initial windowing function which was probably called from the dll that encountered the problem. So I inspect the output from kP and see user32!MessageBoxIndirectA on the call stack which takes a struct. Upon examining the struct I see that the call was bubbled up via odbc32.dll.
1 0:000> dt tagMSGBOXPARAMSA 0x0012fc4c
2 odbc32!tagMSGBOXPARAMSA
3 ...
It turns out that the root cause in this case was the result of datasource issues.