I already touched this topic a while ago, but since it's an important part of the debugging process (and your debugging techniques may vary a lot, depending if you have or not good symbols for your dump) I though would be a good idea to give some more details. And just to jump start on the topic, here's something I learnt a while ago after wasting a few hours typing commands and looking at inconsistent results... debugging with the wrong symbols could be much worse than debugging with no symbols at all.
You can think of symbol files basically as small databases, files which contain source line information, data types, variables, functions and everything else needed to provide names for all lines of code, instead of hexadecimal addresses. They usually have .pdb (or .dbg) extension, and are matched with the actual executable code using an internal timestamp, so it's very important to generate symbols every time you build the application, also for release builds. If you'll ever have a problem with your live application and you'll need to debug it, and if you'll not have the matching symbols (matching means the symbols obtained from the same exact build of the dlls you put in production) you could be in troubles... Building once again the application to create the symbol files, even without changing your code does not really help because the timestamp will not match, and WinDbg will complain about missing symbols anyway...
Note that in Visual Studio if you set "Release" in the "Standard" toolbar, the "Generate debugging information" checkbox will automatically be cleared, so remember to go to the Project properties and flag it again before rebuilding the project.
Since in ASP.NET 2.0 we have a new default page architecture, there is no real need to create the .pdb files unless you're using the "Web Application Project" template which comes with the Service Pack 1 for Visual Studio 2005, in which case you can find it from the project properties, "Compile" tab.
The same applies for Visual Studio 2008.
When you open a dump within WinDbg and you type in a command for example to inspect the call stack, the debugger will start looking for matching symbols to show you an output as detailed as possible: but how does it decide where to look for those symbols? It will use the path(s) specified in the "Symbol Search Path" dialog you can find under the File menu (CTRL+S is the keyboard shortcut.
Here you can specify the symbol servers where WinDbg can download the symbols from, and of course you can use has many server more than one server at a time; WinDbg will simply access those servers in the same order you put in the search path, and it goes through the end of the list until it finds a match.
Microsoft has a public symbol server accessible through the Internet which stores public .pdb files for almost all of our products: http://msdl.microsoft.com/download/symbols.
I work a lot with WinDbg and memory dumps in my daily job, also with my laptop when not connected to the Internet or corporate network, so I need to be able to debug while offline and in any case I don't want to waste time waiting for the tool to download the same symbols over and over again, through one dump to the other... for this reason it's also possible to create a local symbol cache (also referred as downstream store) on your hard disk. When looking for symbols, WinDbg will first of all check your local cache and if the match is found there no additional check is done against the other symbol servers, while if the match is not found WinDbg goes on as usual until it finds the right match; in this case it downloads the .pdb and stores it in your local symbol cache, so that next time it will be readily available for you.
The symbol path is a string composed of multiple directory paths, separated by semicolons. For each directory in the symbol path, the debugger will look in three directories: for instance, if the symbol path includes the directory c:\MyDir, and the debugger is looking for symbol information for a dll, the debugger will first look in c:\MyDir\symbols\dll, then in c:\MyDir\dll, and finally in c:\MyDir. It will repeat this for each directory in the symbol path. Finally, it will look in the current directory, and then the current directory with \dll appended to it. (The debugger will append dll, exe, or sys, depending on what binaries it is debugging.)
Here is a sample symbol path:
The above is actually the symbol path (a bit simplified) I use on my machines: as you can see I have a local cache in C:\Symbols; if I've never downloaded a particular symbol before, WinDbg does to an internal share were we have full symbols, and if still unsuccessful I finally give a try to the public Microsoft symbol server on the Internet. If you include two asterisks in a row where a downstream store would normally be specified, then the default downstream store is used. This store will be located in the sym subdirectory of the home directory. The home directory defaults to the debugger installation directory; this can be changed by using the !homedir extension. If the DownstreamStore parameter is omitted and no extra asterisk is included (i,e. if you use srv with exactly one asterisk or symsrv with exactly two asterisks) then no downstream store will be created and the debugger will load all symbol files directly from the server, without caching them locally. Note that If you are accessing symbols from an HTTP or HTTPS site, or if the symbol store uses compressed files, a downstream store is always used. If no downstream store is specified, one will be created in the sym subdirectory of the home directory.
The symbol server does not have to be the only entry in the symbol path. If the symbol path consists of multiple entries, the debugger checks each entry for the needed symbols; moreover the symbol path can contain several directories or symbol servers, separated by semicolons. This allows you to locate symbols from multiple locations (or even multiple symbol servers). If a binary has a mismatched symbol file, the debugger cannot locate it using the symbol server because it checks only for the exact parameters. However, the debugger may find a mismatched symbol file with the correct name, using the traditional symbol path, and successfully load it; in this case it's important to know if our symbols matches (see the next topic).
You can set the symbol path in advance once for all within WinDbg: open an empty instance, press CTRL+S, type in the path, clock "Ok" on the dialog and close WinDbg, accepting to save the workspace if prompted to do so (next time you'll open WinDbg the value will still be there). Or you can use the .sympath command within WinDbg with a dump open.
Another option is to set the system wide variable _NT_SYMBOL_PATH (the syntax is still the same), used by debuggers like WinDbg or also from adplus directly when it captures the dump.
The same principle applies to the Visual Studio debugger (also have a look at the article http://support.microsoft.com/kb/311503/en-us):
Looking at a call stack sometimes it's clear you're having a problem with unmatched symbols because WinDbg tells you something like:
1: ChildEBP RetAddr
2: 0012f6dc 7c59a2d1 NTDLL!NtDelayExecution(void)+0xb
3: 0012f6fc 7c59a29c KERNEL32!SleepEx(unsigned long dwMilliseconds = 0xfa, int bAlertable = 0)+0x32
4: *** ERROR: Symbol file could not be found. Defaulted to export symbols for aspnet_wp.exe -
5: 0012f708 00442f5f KERNEL32!Sleep(unsigned long dwMilliseconds = 0x444220)+0xb
6: WARNING: Stack unwind information not available. Following frames may be wrong.
7: 0012ff60 00444220 aspnet_wp+0x2f5f
8: 0012ffc0 7c5989a5 aspnet_wp!PMGetStartTimeStamp+0x676
9: 0012fff0 00000000 KERNEL32!BaseProcessStart(<function> * lpStartAddress = 0x004440dd)+0x3d
Unfortunately could happen to not be so lucky, and you'll find yourself wondering if the stack you are looking at is genuine or there are some small (or maybe even not so small) inconsistencies which may lead you down to a completely wrong path. In such cases, you can first of all use the lm command to find which .pdb files have been loaded:
1: kernel32 (pdb symbols) .sympath SRV\kernel32.pdb\CE65FAF896A046629C9EC86F626344302\kernel32.pdb
2: ntdll (pdb symbols) .sympath SRV\ntdll.pdb\36515FB5D04345E491F672FA2E2878C02\ntdll.pdb
3: shell32 (deferred)
4: user32 (deferred)
As you can see in the example above, two symbols were loaded (for kernel32.dll and ntdll.dll), while shell32.dll and user32.dll were not part of the stack analyzed, so WinDbg has not loaded yet (deferred) their symbols. A bad match will look like the following:
1: ntdll M (pdb symbols) .sympath SRV\ntdll.pdb\36515FB5D04345E491F672FA2E2878C02\ntdll.pdb
Notice the "M" highlighted in red (could also be a "#" pound sign)? That stands or "mismatch", and indicates there is a problem with that particular module (search for "Symbol Status Abbreviations" in WinDbg help for further details). Alternatively you can use the !sym noisy and .reload command to reload symbols verbosely to have a detailed output. Look for "Symbols files and paths - Overivew" in WinDbg help for more details.
I'm not sure why this happens, and especially why I had this problem only with ntdll.dll (and its .pdb): I was not able to get the proper stack even with a matching symbol (and I checked more than once to be really sure)... until I got the idea to delete the ntdll.pdb folder in my local cache (if you have a dump open you must first unload the symbol from WinDbg or the file will be locked: use the .reload /u <module_name> command), then run a .reload /f <module_name> (/f forces immediate symbol load) and let WinDbg to download it again... this usually does the trick and I finally get the correct stack.
It's not impossible, but it's harder than debugging with matching symbols; the main difference is that you'll not be able to see method names, variable names etc... and generally speaking the stack will be less easily readable. To give you a quick example, here is an excerpt of the stack of a very simple application I wrote for test (it has just a button which sets the text of a label to the DateTime.Current.ToString()):
1: 5 Id: 10d4.1204 Suspend: 1 Teb: 7ffd7000 Unfrozen
2: ldEBP RetAddr
3: NING: Frame IP not in any known module. Following frames may be wrong.
4: 9f524 71a6b7f8 0x7c90eb94
5: 9fa0c 03490657 0x71a6b7f8
6: WARNING: Unable to verify checksum for System.dll
7: ERROR: Module load completed but symbols could not be loaded for System.dll
8: 9fa40 7a603543 CLRStub[StubLinkStub]@3490657(<Win32 error 318>)
9: a8240 032908ff System!System.Net.Sockets.Socket.Accept(<HRESULT 0x80004001>)+0xc7
10: ERROR: Module load completed but symbols could not be loaded for WebDev.WebHost.dll
11: 9fab0 7940a67a WebDev_WebHost!Microsoft.VisualStudio.WebHost.Server.OnStart(<HRESULT 0x80004001>)+0x27
12: WARNING: Unable to verify checksum for mscorlib.dll
13: ERROR: Module load completed but symbols could not be loaded for mscorlib.dll
14: bd1b4 7937d2bd mscorlib!System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(<HRESULT 0x80004001>)+0x1a
15: bd1b4 7940a7d8 mscorlib!System.Threading.ExecutionContext.Run(<HRESULT 0x80004001>)+0x81
16: 9fae0 7940a75c mscorlib!System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(<HRESULT 0x80004001>)+0x44
17: 32010 79e79dd3 mscorlib!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(<HRESULT 0x80004001>)+0x60
18: 9fb04 79e79d57 0x79e79dd3
19: 9fb84 79f71cba 0x79e79d57
20: 9fba4 79f71c64 0x79f71cba
21: 9fc08 79f71cf3 0x79f71c64
22: 9fc3c 7a0b0896 0x79f71cf3
23: 9fc9c 79f7ba4f 0x7a0b0896
24: 9fcb0 79f7b9eb 0x79f7ba4f
25: 9fd44 79f7b90c 0x79f7b9eb
26: 9fd80 79ef9887 0x79f7b90c
27: 9fda8 79ef985e 0x79ef9887
28: 9fdc0 7a0a32da 0x79ef985e
29: 9fe28 79ef938f 0x7a0a32da
30: 9fe94 79f7be67 0x79ef938f
31: 9ffb4 7c80b683 0x79f7be67
32: 9ffec 00000000 0x7c80b683
with matching symbols:
3: 9f4e4 7c90e9c0 ntdll!KiFastSystemCallRet
4: 9f4e8 71a54033 ntdll!ZwWaitForSingleObject+0xc
5: 9f524 71a6b7f8 mswsock!SockWaitForSingleObject+0x1a0
6: 9f9bc 71ac0e2e mswsock!WSPAccept+0x21f
7: 9f9f0 71ac103f ws2_32!WSAAccept+0x85
8: 9fa0c 03490657 ws2_32!accept+0x17
9: WARNING: Unable to verify checksum for System.ni.dll
10: 9fa40 7a603543 CLRStub[StubLinkStub]@3490657(<Win32 error 318>)
11: a8240 032908ff System_ni!System.Net.Sockets.Socket.Accept(<HRESULT 0x80004001>)+0xc7
12: 9fab0 7940a67a WebDev_WebHost!Microsoft.VisualStudio.WebHost.Server.OnStart(<HRESULT 0x80004001>)+0x27
13: WARNING: Unable to verify checksum for mscorlib.ni.dll
14: bd1b4 7937d2bd mscorlib_ni!System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(<HRESULT 0x80004001>)+0x1a
15: bd1b4 7940a7d8 mscorlib_ni!System.Threading.ExecutionContext.Run(<HRESULT 0x80004001>)+0x81
16: 9fae0 7940a75c mscorlib_ni!System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(<HRESULT 0x80004001>)+0x44
17: 32010 79e79dd3 mscorlib_ni!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(<HRESULT 0x80004001>)+0x60
18: 9fb04 79e79d57 mscorwks!CallDescrWorker+0x33
19: 9fb84 79f71cba mscorwks!CallDescrWorkerWithHandler+0xa3
20: 9fba4 79f71c64 mscorwks!DispatchCallBody+0x1e
21: 9fc08 79f71cf3 mscorwks!DispatchCallDebuggerWrapper+0x3d
22: 9fc3c 7a0b0896 mscorwks!DispatchCallNoEH+0x51
23: 9fc9c 79f7ba4f mscorwks!QueueUserWorkItemManagedCallback+0x6c
24: 9fcb0 79f7b9eb mscorwks!Thread::DoADCallBack+0x32a
25: 9fd44 79f7b90c mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
26: 9fd80 79ef9887 mscorwks!Thread::ShouldChangeAbortToUnload+0x30a
27: 9fda8 79ef985e mscorwks!Thread::ShouldChangeAbortToUnload+0x33e
28: 9fdc0 7a0a32da mscorwks!ManagedThreadBase::ThreadPool+0x13
29: 9fe28 79ef938f mscorwks!ManagedPerAppDomainTPCount::DispatchWorkItem+0xdb
30: 9fe3c 79ef926b mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0xaf
31: 9fe94 79f7be67 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x223
32: 9ffb4 7c80b683 mscorwks!Thread::intermediateThreadProc+0x49
33: 9ffec 00000000 kernel32!BaseThreadStart+0x37
The difference is quite obvious... The WinDbg help file also gives a few hints:
The easiest thing you could do is use symstore.exe (you'll find it in WinDbg/adplus installation folder) with a command like the following:
symstore add /f c:\temp\SymbolsTest\Bin\*.pdb /s c:\symbols /t "Symbols Test"
symstore add /f c:\temp\SymbolsTest\Bin\*.pdb /s c:\symbols /t "Symbols Test"
Let's have a quick look at the syntax:
Symstore will create a structure similar to the following:
Also note the "0000Admin" folder created by symstore, which contains one file for each transaction (every "add" or "delete" operation is recorded as a transaction), as well as the logs server.txt and history.txt; the former contains a list of all transactions currently on the server, while the latter contains a chronological history of all transactions run on the machine. For further information you can see the "Using SymStore" topic in WinDbg help (debugger.chm).
While it is possible to debug without symbols (this is true especially for managed code), remember that your life could be much easier if you (and your customers) will take care of your symbols and will generate them every time the application will be rebuilt, also in release mode.
I deliberately simplified the argument (I just wanted to share what I learnt in my daily job and give some quick tips to get started), much more could be said and if you're interested I encourage you to read the "Symbols" section in the WinDbg help, or search the Internet where you'll find good blog posts/articles on this subject
PingBack from http://msdnrss.thecoderblogs.com/2007/09/23/why-should-we-care-about-symbols/
Thanks again for good article.
When you say,
"You can think of symbol files basically as small databases, files which contain source, line information, data types, variables, functions..."
I doubt the statement. Does symbol files really contain Source?? (this is my favorite interview question..) I think there is extra "," between 'source' and 'line information'.
Also, I have never tried to use,
Instead, I use,
Not sure if three * in the symbol path will work properly, because I have never tried that.
Hi Jigar, that was actually a typo (corrected), thanks! :-)
A more detailed list of what you can expect a symbol file to contain is:
About the symbol path, I actually use the one with two "*", it means that if I don't have the .pdb in my local cache, and it's neither in \\internalshare but it's actually found in the Microsoft public store, it will first be copied down to \\internalshare and then in my local store too. The path you are using means what if you download the symbol from Microsoft public store it will be copied directly in your local store, but not in \\internalshare, that's the difference (to be honest I've not tested it, but that's what the docs say ;-))
Well this is a nice article.
And i am facing the same problem as u have mentioned above for debugging.
I have given all the paths i think ,in symbol link but still it gives symbol not found.As a result I am not able to place a break point and debug my application.
I have even downloaded symbols from msdn site as well whatever files with .pdb extention are available in debug folder i have provided the link of that as well but still the problem persists.
Kindly provide some assistance.
Hi Ravi, do you mean Windbg is not able to locate matching symbols? If what's missing is about a Microsoft component it should be stored at least in the public symbols store (check carefully the URL: it's *msdl*, not "msdn", it's a common mistake).
If you already tried the various combinations of reload commands, could you check what happens on a different machine? Maybe that's a problem with that specific one...
Also have a look at "Installing Windows Symbols Files" in the Windbg help for details about how to get the entire set of symbols at once (download, on CD etc...) and install them locally on your dev machine; also have a look at SymCheck.exe in the docs, it may be useful.
I've just finished writing up an e-mail for some new people in my team about starting Debugging and the