If broken it is, fix it you should

Using the powers of the debugger to solve the problems of the world - and a bag of chips    by Tess Ferrandez, ASP.NET Escalation Engineer (Microsoft)

Why do I get weird function names on my stack? (a discussion on symbols)

Why do I get weird function names on my stack? (a discussion on symbols)

Rate This
  • Comments 20

Symbols can contain information about global variables, local variables, function names, parameters, structures and source line numbers.

There are 3 types of symbols, export symbols, pdb symbols (public symbols) and private pdb symbols (private symbols).

The export symbols are part of the dll itself. For example ntdll.dll and kernel32.dll expose a big part of their functions as export symbols so that they can be called as API’s, but most dll’s that you find in a process have a very small set of exported symbols. Generally export symbols don’t contain parameter info for the functions and since very few functions are exposed this way you can’t really relay on the validity of the stacks when you have only export symbols.

Public symbols contain some basic symbols such as function names and global variables, but again, not all function names are exposed in public symbols. The developer of the dll chooses what to expose as public symbols and thus he/she can hide anything that they feel would give away too much information about the implementation.

Private symbols contain pretty much everything listed in the first paragraph

When debugging, symbols are matched up with the respective dlls or exe’s by a GUID linking the dll/exe to a symbol file (thanks Pavan for the correction). This means that if you have multiple ntdll.pdb’s in your symbol search path, the debugger will know which one corresponds to your particular version of ntdll.dll. The search path is given by .sympath, and apart from what is listed in your sympath the debugger will also look in the directory where the dll is loaded from as well as anything in the paths given in the environment variable _NT_SYMBOL_PATH.

So… what happens if you have the wrong symbols?

Let’s look at this stack with public symbols for mscorsvr.dll: (sorry about the formatting)

  54  Id: 62c.1590 Suspend: 1 Teb: 7ffa2000 Unfrozen
ChildEBP RetAddr  Args to Child              
1212ef44 7c59a030 00000090 00000000 1212ef64 ntdll!NtWaitForSingleObject+0xb [i386\usrstubs.asm @ 2004]
1212ef6c 7c57b3db 00000090 00009c40 00000000 kernel32!WaitForSingleObjectEx+0x71 [D:\nt\private\windows\base\client\synch.c @ 1309]
1212ef7c 791b578b 00000090 00009c40 00000000 kernel32!WaitForSingleObject+0xf [D:\nt\private\windows\base\client\synch.c @ 1217]
1212efa0 791dbe6e 00000000 00000000 00000000 mscorsvr!ThreadpoolMgr::WorkerThreadStart+0x3a
1212ffb4 7c57b388 0d406838 00000002 00000000 mscorsvr!ThreadpoolMgr::intermediateThreadProc+0x44
1212ffec 00000000 791dbe2d 0d406838 00000000 kernel32!BaseThreadStart+0x52 [D:\nt\private\windows\base\client\support.c @ 460]

We are creating a worker thread and then sit and wait for work

Let’s look at the same stack with export symbols for mscorsvr:

0:054> kb
ChildEBP RetAddr  Args to Child              
WARNING: Stack unwind information not available. Following frames may be wrong.
1212ef6c 7c57b3db 00000090 00009c40 00000000 ntdll!NtWaitForSingleObject+0xb
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for mscorsvr.dll - 
1212efa0 791dbe6e 00000000 00000000 00000000 kernel32!WaitForSingleObject+0xf
1212ffb4 7c57b388 0d406838 00000002 00000000 mscorsvr!GetCompileInfo+0x8e99
1212ffec 00000000 791dbe2d 0d406838 00000000 kernel32!lstrcmpiW+0xb7

The debugger is nice and tells us that we couldn’t find the symbol file for mscorsvr, but yet it gives us a function name (from the export symbols) so it appears that we’re calling some function called GetCompileInfo, hmm… very strange, why did it pick that name?

If we were to list the symbols (export symbols for mscorsvr) we would get a list that looks like this

0:054> x mscorsvr!*
791b0000 mscorsvr!Ordinal73 = 
791b0000 mscorsvr!Ordinal76 = 
791b0000 mscorsvr!Ordinal77 = 
791b0000 mscorsvr!Ordinal75 = 
791b0000 mscorsvr!Ordinal78 = 
791b0000 mscorsvr!Ordinal71 = 
791b0000 mscorsvr!Ordinal79 = 
791b0000 mscorsvr!Ordinal74 = 
791b0000 mscorsvr!Ordinal72 = 
791d2fd5 mscorsvr!GetCompileInfo = 
791e0920 mscorsvr!GetAssemblyMDImport ()
…

So we are executing something at address 791dbe6e (791d2fd5+0x8e99), if we take a peak at the symbols with public symbols loaded, we would see that that address landed inside the intermediateThreadProc function (+0x44 to be exact)

791bbb78 mscorsvr!SimpleRWLock::LeaveRead = 
793eaf50 mscorsvr!ExecutionManager::m_dwReaderCount = 
791dbe2d mscorsvr!ThreadpoolMgr::intermediateThreadProc = 
793035c5 mscorsvr!VtBoolMarshaler::UnmarshalComToNativeIn = 
791f8e0d mscorsvr!LayoutClassPtrMarshaler::ClearNativeTemp = 

So in essence, the reason it picked the name GetCompileInfo was that that was the last symbol it could find, preceding the address that we were executing at.

You can see how this can easily get confusing. Oh and btw, not only did it give us a completely wrong function name, but if you take a closer look at the two stacks, we lost a whole function call on the stack with the export symbols (The call to WorkerThreadStart). The reason for this is that the function we were supposed to look at was at address 791b578b and this address is located before our first exported function so it can’t even resolve it to something fake.

How do you know if your symbols are good?

For starters, just by looking at the stacks we can see that the function names seem pretty weird. It doesn’t make sense that a thread would start with lstrcmpiW, they usually start with BaseThreadStart or something to that effect. The second thing that gives it away just by looking at the stack is that we were supposed to be 0x8e99 instructions in to the function GetCompileInfo, that is 36505 instructions, wow, that would be a mighty long function.

The more definite way of knowing would be to run lmv on the executable. (notice the extra m before mscorsvr for match pattern)

0:054> lmv mmscorsvr
start    end        module name
791b0000 79418000   mscorsvr   (export symbols)       mscorsvr.dll
    Loaded symbol image file: mscorsvr.dll
    Image path: C:\WINNT\Microsoft.NET\Framework\v1.1.4322\mscorsvr.dll
    Image name: mscorsvr.dll
    Timestamp:        Thu Jul 15 09:26:32 2004 (40F631A8)
    CheckSum:         002664D4
    ImageSize:        00268000
    File version:     1.1.4322.2032
    Product version:  1.1.4322.2032
    File flags:       20 (Mask 3F) Special
    File OS:          4 Unknown Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04e4
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft .NET Framework
    InternalName:     MSCORSVR.DLL
    OriginalFilename: mscorsvr.dll
    ProductVersion:   1.1.4322.2032
    FileVersion:      1.1.4322.2032
    FileDescription:  Microsoft .NET Runtime Common Language Runtime - Server
    LegalCopyright:   Copyright © Microsoft Corporation 1998-2002. All rights reserved.
    LegalTrademarks:  Microsoft® is a registered trademark of Microsoft Corporation. Windows(TM) is a trademark of Microsoft Corporation
    Comments:         Microsoft .NET Runtime Common Language Runtime - Server

Lmv will give you a lot of good information about the dll, but in this particular case what we are interested is what is printed after the module name, i.e export symbols in this case. (You can also get this for all modules by running lm)

One last comment on this: As you may have noticed, going to the public Microsoft symbols server you sometimes get public symbols and sometimes end up with export symbols for mscorsvr.dll for example. This is because some cases like certain hotfixes and private builds have not been uploaded to the public symbol server.

Lazy symbol loading and downloading symbols for offline debugging

If you run lm you will notice that a lot of the dll’s and exe’s will have their symbols listed as deferred. This is because the debugger doesn’t actually load all the symbols when the debugger starts (as this would take a looooooooong time especially if you have a remote symbol store). Symbols are loaded on an as needed bases, i.e. when you run kb (to list the stacks) or run x to examine the symbols for a particular executable, or any other command that requires symbol lookup.

Sometimes though, it is useful to download all the symbols for a process, a common example would be if your application is running on a machine with no internet access and you want to be able to live debug it.

In this case what you can do is this:

  1. Take a dump of the process when all modules are loaded and open it on a machine with internet access.
  2. Set the sympathy to srv*c:\myappsymbols*http://msdl.microsoft.com/download/symbols;
  3. run .reload /i /f to force load all the symbols for the dump

This will download all the symbols available on Microsofts public symbol server that match your dlls into the c:\myappsymbols and then you can simply copy this folder over to the machine without internet access and set the symbolpath there to srv*c:\myappsymbols*c:\myappsymbols;c:\folderwithanyadditionalsymbols.

But… some of the functions on my stack only show up as addresses, what is up with that?

This really doesn’t have anything to do with symbols but I thought it was worth mentioning anyways. In the callstack below we can see one function call only listed by it’s address… 0xd44334a, the reason for this is that it is a managed function which doesn’t have a native translation, and after all windbg is merely a native debugger. You can run !clrstack or !ip2md on the address to get the .net function name.

0:045> kb
ChildEBP RetAddr  Args to Child              
1182ecd0 7c59c5b7 7920d6a7 00000010 0f39c800 ntdll!NtYieldExecution+0xb [i386\usrstubs.asm @ 2108]
1182ecd4 7920d6a7 00000010 0f39c800 00000010 kernel32!SwitchToThread+0x6 [D:\nt\private\windows\base\client\thread.c @ 2611]
1182ed00 791c0283 0f39c800 00000010 00000000 mscorsvr!gc_heap::allocate_more_space+0xe3
1182ef28 791b6930 0f39c800 00000010 00000000 mscorsvr!GCHeap::Alloc+0x7b
1182ef3c 791c0359 00000010 00000000 00000000 mscorsvr!Alloc+0x3a
1182ef5c 791c03a7 0d421928 1182efec 062eff1c mscorsvr!FastAllocateObject+0x25
1182efc8 0d44334a 00000000 1182f01c 1182f040 mscorsvr!JIT_NewFast+0x2c
WARNING: Frame IP not in any known module. Following frames may be wrong.
1182effc 791b33ec 1182f114 791b3c96 1182f050 0xd44334a
1182f004 791b3c96 1182f050 00000000 1182f028 mscorsvr!CallDescrWorker+0x30
1182f114 791b5612 00bf1bdb 79b7a000 00000004 mscorsvr!MethodDesc::CallDescr+0x1b8
1182f1d0 791b56c6 79bf1bdb 79b7a000 79bd1d16 mscorsvr!MethodDesc::CallDescr+0x4f
1182f1f8 791b5c40 1182f210 0ba57968 0f39c7c8 mscorsvr!MethodDesc::Call+0x97
1182f2ac 791b5b97 0f3a6480 00000001 00000000 mscorsvr!AddTimerCallbackEx+0x165
1182f2c0 791b4de8 1182f374 791b3cf4 ffffffff mscorsvr!AddTimerCallback_Wrapper+0x13
1182f308 791b5ae5 0ba57968 791b5b84 1182f374 mscorsvr!Thread::DoADCallBack+0x5c
1182f3c4 791b59ce 0f3a6480 00000001 00000001 mscorsvr!AddTimerCallbackEx+0x165
1182f3d8 791b59a5 0f3a6480 00000001 791b5996 mscorsvr!AddTimerCallback+0x10
1182f3ec 791b5802 0f3a2230 0f3b6bf0 793ea988 mscorsvr!ThreadpoolMgr::AsyncTimerCallbackCompletion+0xf
1182f400 791b57c1 0f3b6bf0 00000000 791b5751 mscorsvr!ThreadpoolMgr::ExecuteWorkRequest+0x19
1182f420 791dbe6e 00000000 00000000 00000000 mscorsvr!ThreadpoolMgr::WorkerThreadStart+0x129


0:045> !clrstack
Loaded Son of Strike data table version 5 from "C:\WINNT\Microsoft.NET\Framework\v1.1.4322\mscorsvr.dll"
Thread 45
ESP         EIP       
0x1182efa4  0x77f88fe3 [FRAME: HelperMethodFrame] 
0x1182efd0  0x0d44334a [DEFAULT] [hasThis] Void System.Timers.Timer.MyTimerCallback(Object)
0x1182f2cc  0x791b33ec [FRAME: ContextTransitionFrame]

How about symbols for managed dlls? Do I need them?

Technically for debugging with windbg the answer is that you would need them very seldom or never. If you do have managed symbols and the dlls are compiled in debug mode you can get line numbers and source files. You can also get some parameter info when using !clrstack –p. But all in all, the extra “stuff” you get from managed symbols is very little when using a native debugger like windbg.

Over and out…  

 





  • Wow thank you! This stuff is just mind blowing. How did you learn all that?
  • this blog is one (if not) the best debugging blogs I'ever visited. -- John - Texas
  • I have talked about a number of different hang/performance issues in my posts. This post is a generic

  • Wow. I was trying to figure out how I was going to load symbols for Windows DLLs that don't seem to match the symbol packs while remote debugging without being able to access the internet to get to the symbol server (stupid VPN software redirects all traffic except DHCP and the VPN connection over the VPN, and the customer site does not allow internet access.) I'm debugging using Visual Studio, and was not aware that I could create a dump file using it. I'm currently waiting for the dump file to transfer across the VPN so I can load it while I have internet access. I don't see why this wouldn't work, though. Thanks!

    (found this blog entry via a google search, fwiw)

  • Apparently, I can create the dump using VS, but in order to download the symbols, I have to load the dump into WinDbg and do the force reload command given above. Once I do that, everything works. Thanks again!

  • Sorry pelase :

    Wrong cdteagory...

    wippbe careful

  • .NET Debugging Demos This is a series of debugging demos aimed to help you get some hands on experience

  • When you debug .net applications you will sometimes get error messages in windbg. Here are a few of the

  • If you have read any of my posts you have probably noticed that I am very partial to windbg and the debugging

  • .NET调试实例

    这是一个系列的调式实例,目的是为了帮助你在调式.NET应用程序中最常见的挂起(Hang)、性能(performance)、内存(memory)和系统崩溃(crash)方面获得一些...

  • The debugger matches the symbols not by the timestamp and checksum but by a GUID that links the dll and pdb. This GUID is generated by the compiler and is inserted into the dll and pdb headers.

    Thanks.

  • 0:054> lmv mmscorsvr

    shoulde be

    0:054> lmv m mscorsvr

  • Yanglei,  you can do it either way

  • I was wondering what steps are need for offline dump analysis for .net code. I always end up xcopying the tools to the prod server because when I take a crash dump and copy it local the managed functions are gone.

    Example !clrstack from server

    0:031> !clrstack

    OS Thread Id: 0x754 (31)

    ESP       EIP    

    1ed6ed94 7939f908 System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].FindEntry(System.__Canon)

    1ed6edb0 7939f510 System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].get_Item(System.__Canon)

    1ed6edbc 1ee44048 MarketingBusinessLayer.DefaultSettingsConfigHelper.BuildDefaultPathFromConfigWithSiteURL(System.String)

    1ed6edcc 1ee43f79 MarketingBusinessLayer.DefaultSettingsConfigHelper.BuildFolderPathForSelectedFile(System.String, System.String, System.String)

    1ed6ede4 1fcc16b9 ControlFactory.BuildUserControl(System.Xml.XPath.XPathNavigator)

    1ed6edf4 1fcc1525 _Default.Page_Load(System.Object, System.EventArgs)

    1ed6ee30 698a1928 System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr, System.Object, System.Object, System.EventArgs)

    1ed6ee40 6891d73f System.Web.Util.CalliEventHandlerDelegateProxy.Callback(System.Object, System.EventArgs)

    1ed6ee54 687ca5a4 System.Web.UI.Control.OnLoad(System.EventArgs)

    1ed6ee64 687ca5f0 System.Web.UI.Control.LoadRecursive()

    1ed6ee78 687dbbf6 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)

    1ed6f030 687db521 System.Web.UI.Page.ProcessRequest(Boolean, Boolean)

    1ed6f060 687db487 System.Web.UI.Page.ProcessRequest()

    1ed6f098 687db3a7 System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)

    1ed6f0a0 687db33a System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)

    1ed6f0b4 1fcc0a95 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)

    1ed6f0b8 686888df System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()

    1ed6f0ec 6865a071 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)

    1ed6f12c 6865a39b System.Web.HttpApplication.ResumeSteps(System.Exception)

    1ed6f174 6865912d System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object)

    1ed6f190 6865e0bd System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest)

    1ed6f1c4 6865dd72 System.Web.HttpRuntime.ProcessRequestNoDemand(System.Web.HttpWorkerRequest)

    1ed6f1d0 6865c447 System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr, Int32)

    1ed6f380 79f1ef33 [ContextTransitionFrame: 1ed6f380]

    1ed6f3d0 79f1ef33 [GCFrame: 1ed6f3d0]

    1ed6f528 79f1ef33 [ComMethodFrame: 1ed6f528]

    then when I copy the dump file local

    0:031> !clrstack

    OS Thread Id: 0x754 (31)

    ESP       EIP    

    1ed6ed94 7939f908

    1ed6edb0 7939f510

    1ed6edbc 1ee44048

    1ed6edcc 1ee43f79

    1ed6ede4 1fcc16b9

    1ed6edf4 1fcc1525

    1ed6ee30 698a1928

    1ed6ee40 6891d73f

    1ed6ee54 687ca5a4

    1ed6ee64 687ca5f0

    1ed6ee78 687dbbf6

    1ed6f030 687db521

    1ed6f060 687db487

    1ed6f098 687db3a7

    1ed6f0a0 687db33a

    1ed6f0b4 1fcc0a95

    1ed6f0b8 686888df

    1ed6f0ec 6865a071

    1ed6f12c 6865a39b

    1ed6f174 6865912d

    1ed6f190 6865e0bd

    1ed6f1c4 6865dd72

    1ed6f1d0 6865c447

    1ed6f380 79f1ef33 [ContextTransitionFrame: 1ed6f380]

    1ed6f3d0 79f1ef33 [GCFrame: 1ed6f3d0]

    1ed6f528 79f1ef33 [ComMethodFrame: 1ed6f528]

    it seems as though windbg needs the actual .net exe and dll files to get the info from because none of the .net code was built debug. Is offline dump analysis of sites that use the gac and temp dlls possible / practical without copying a bunch of files along with the dump file?

  • it looks like a problem with sos where you dont have the proper symbols for mscordacwks on the client.  Do you have the same framework version on the client and server?

Page 1 of 2 (20 items) 12
Leave a Comment
  • Please add 6 and 7 and type the answer here:
  • Post