Once you have collected a dump file, to analyse it you need to use a tool called WinDbg. In this post I am going to explain how to set up WinDbg so it's ready to debug a memory dump taken from a Dynamics AX process. If you're not sure how to create a dump file, just take a look at the post below, scroll down to the subtitle "What do I need to create a Crash Dump?" and you'll find what you need there:

http://blogs.msdn.com/b/emeadaxsupport/archive/2010/05/12/possibilities-to-create-memory-dumps-from-crashing-processes.aspx

Getting WinDbg set up is a fairly straightforward task, just download and install the Windows debugging tools from the link below. You might need to install both the 32bit and 64 bit versions – if you are looking at dump files from a 32bit operating system then use the 32bit tools, and if it’s from a 64bit operating system use the 64 bit tools.

32-bit OS: http://msdn.microsoft.com/en-us/windows/hardware/gg463016.aspx
64-bit OS: http://msdn.microsoft.com/en-us/windows/hardware/gg463012.aspx

If you’re not sure which OS a particular dump file came from (and it's always good to check just to be sure) then when you open a dump file, with either version of WinDbg it will tell you which operating system it was taken from, here’s an example, with the important part highlighted in yellow:


Microsoft (R) Windows Debugger Version 6.11.0001.404 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\Temp\Ax32Serv.exe.4408.dmp]
User Mini Dump File with Full Memory: Only application data is available


WARNING: Whitespace at end of path element
Symbol search path is: http://msdl.microsoft.com/download/symbols

 

Executable search path is:
Windows 7 Version 7600 UP Free x64
Product: Server, suite: Enterprise TerminalServer SingleUserTS
Machine Name:
Debug session time: Thu Dec 30 11:26:55.000 2010 (GMT+0)
System Uptime: 0 days 23:42:28.537
Process Uptime: 0 days 0:21:20.000
................................................................
................
Loading unloaded module list
......
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(1138.c3c): Stack overflow - code c00000fd (first/second chance not available)
ntdll!NtWaitForSingleObject+0xa:
00000000`779efd9a c3              ret

Before you start to use WinDbg you also need to configure the symbol path – just go to file->symbol file path and the path you need to enter for the Microsoft public symbol server is:

http://msdl.microsoft.com/download/symbols

If you haven’t come across the concept of symbols before, then a short explanation is that the symbols are used to decode the information held in the memory dump file which allows you to see the function names in the call stack, to give an example of what you might see with and without symbols:

With symbols:


0:010> kp
Child-SP          RetAddr           Call Site
00000000`02832f48 00000000`77a65e02 ntdll!NtWaitForSingleObject+0xa
00000000`02832f50 00000000`77a65fa5 ntdll!RtlReportExceptionEx+0x1d2
00000000`02833040 000007fe`fe257899 ntdll!RtlReportException+0xb5
00000000`028330c0 000007fe`fe23e702 rpcrt4!RpcpReportFatalErrorExceptionFilter+0x19
00000000`028330f0 00000000`779b50c8 rpcrt4!Invoke+0x82b
00000000`02833120 00000000`779d52cd ntdll!_C_specific_handler+0x8c
00000000`02833190 00000000`779b5c9c ntdll!RtlpExecuteHandlerForException+0xd
00000000`028331c0 00000000`779efcb8 ntdll!RtlDispatchException+0x3cb
00000000`028338a0 00000000`779f2880 ntdll!KiUserExceptionDispatcher+0x2e
00000000`02833e60 00000000`779f283c ntdll!RtlpAllocateHeap+0x30
00000000`02834400 000007fe`fdbc15b5 ntdll!RtlAllocateHeap+0x16c
00000000`02834510 000007fe`fd363007 KERNELBASE!LocalAlloc+0x71
00000000`02834580 000007fe`fd374684 authz!AuthzpAllocateAndInitializeClientContext+0x27
00000000`028345b0 000007fe`fd8a6e8a authz!AuthzInitializeContextFromAuthzContext+0x144
00000000`02834680 000007fe`fe28f41e RpcRtRemote!RpcpDuplicateAuthzContext+0x36
00000000`028346c0 000007fe`fe1d5aa2 rpcrt4!OSF_SCALL::GetAuthorizationContext+0xce
00000000`02834780 00000000`006cff5b rpcrt4!RpcGetAuthorizationContextForClient+0x104
00000000`028347d0 00000000`008d41e2 Ax32Serv!GetRpcLoggedOnUserSid+0x4b
00000000`02834820 00000000`008d5c03 Ax32Serv!verifyCallIdentity+0x22
00000000`02834850 00000000`00638c76 Ax32Serv!verifyRemoteCaller+0x133

Without symbols:


0:010> kp
Child-SP          RetAddr           Call Site
00000000`02832f48 00000000`77a65e02 ntdll!ZwWaitForSingleObject+0xa
00000000`02832f50 00000000`77a65fa5 ntdll!RtlReportSqmEscalation+0xfc2
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for rpcrt4.dll -
00000000`02833040 000007fe`fe257899 ntdll!RtlReportException+0xb5
00000000`028330c0 000007fe`fe23e702 rpcrt4!I_RpcPauseExecution+0xa9
00000000`028330f0 00000000`779b50c8 rpcrt4!I_RpcInitNdrImports+0x1caf2
00000000`02833120 00000000`779d52cd ntdll!_C_specific_handler+0x9c
00000000`02833190 00000000`779b5c9c ntdll!RtlCompareUnicodeString+0xad
00000000`028331c0 00000000`779efcb8 ntdll!RtlTimeToSecondsSince1970+0x62c
00000000`028338a0 00000000`779f2880 ntdll!KiUserExceptionDispatcher+0x2e
00000000`02833e60 00000000`779f283c ntdll!RtlRestoreLastWin32Error+0x9b0
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for KERNELBASE.dll -
00000000`02834400 000007fe`fdbc15b5 ntdll!RtlRestoreLastWin32Error+0x96c
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for authz.dll -
00000000`02834510 000007fe`fd363007 KERNELBASE!LocalAlloc+0x75
00000000`02834580 000007fe`fd374684 authz!AuthzAccessCheck+0x5b7
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for RpcRtRemote.dll -
00000000`028345b0 000007fe`fd8a6e8a authz!AuthzInitializeContextFromAuthzContext+0x144
00000000`02834680 000007fe`fe28f41e RpcRtRemote!DllGetContractDescription+0x387a
00000000`028346c0 000007fe`fe1d5aa2 rpcrt4!NdrServerContextUnmarshall+0xa5e
*** ERROR: Module load completed but symbols could not be loaded for Ax32Serv.exe
00000000`02834780 00000000`006cff5b rpcrt4!RpcGetAuthorizationContextForClient+0xe2
00000000`028347d0 00000000`008d41e2 Ax32Serv+0x2cff5b
00000000`02834820 00000000`008d5c03 Ax32Serv+0x4d41e2
00000000`02834850 00000000`00638c76 Ax32Serv+0x4d5c03

As you can see in the above example, without symbols it is not possible to read the call stack. Each specific version of an application has a unique symbol file – as they related directly to the source code for that application, so if one line of source code is different inside the application then a new symbol file is generated to match it.

You can run the command “lmv max32serv” in WinDbg to find out which kernel version the AOS is running (if it’s a dump taken from an AOS) or “lmv max32” to find out which version a client is from a client dump. The output in Windbg will look like this:


0:010> lmv max32serv
start             end                 module name
00000000`00400000 00000000`00f2f000   Ax32Serv   (no symbols)          
    Loaded symbol image file: Ax32Serv.exe
    Image path: C:\Program Files\Microsoft Dynamics AX\50\Server\DynamicsAx2009\Bin\Ax32Serv.exe
    Image name: Ax32Serv.exe
    Timestamp:        Sat Nov 06 20:20:23 2010 (4CD5B887)
    CheckSum:         00B2252F
    ImageSize:        00B2F000
    File version:     5.0.1500.3761
    Product version:  5.0.1500.3761
    File flags:       0 (Mask 3F)
    File OS:          4 Unknown Win32
    File type:        1.0 App
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft Dynamics AX
    InternalName:     AX32SERV
    OriginalFilename: AX32SERV.EXE
    ProductVersion:   5.0.1500.3761
    FileVersion:      5.0.1500.3761
    FileDescription:  Microsoft Dynamics AX Application Object Server
    LegalCopyright:   © 2008 Microsoft Corporation
    LegalTrademarks:  Microsoft® is a registered trademark of Microsoft Corporation. Windows(TM) is a trademark of Microsoft Corporation.

Sometimes there might be a problem loading symbols for the specific version of AX that you are running – in that case there are some WinDbg commands which can help you, first you can turn on “noisy” symbol loading prompts so that WinDbg will give information about where it is trying to find the symbols and what the result was (like “file not found” or something like this), there are two commands you need in Windbg to do this:

!sym –noisy
.reload


The .reload command will attempt to reload symbols, and the “!sym –noisy” will turn on noisy prompts. You can also force WinDbg to load a mismatched symbol file – if you can’t find the exact matching file for your version, the command to do that is:

.reload –i ax32serv.exe


Now you're ready to do some analysis and extract some interesting information from the dump file, to do that see the following posts:

http://blogs.msdn.com/b/emeadaxsupport/archive/2011/04/10/finding-the-ax-user-that-caused-a-crash.aspx

http://blogs.msdn.com/b/emeadaxsupport/archive/2011/04/10/finding-the-x-call-stack-that-caused-a-crash.aspx

http://blogs.msdn.com/b/emeadaxsupport/archive/2011/04/10/finding-the-ax-user-and-the-x-call-stack-from-a-memory-dump-the-easy-way.aspx

--author: Tariq Bell
--editor: Tariq Bell
--date: 10/04/2011