Why should we care about symbols?

I already touched this topic a while ago, but since it's an important part of the debugging process (and your debugging techniques may vary a lot, depending if you have or not good symbols for your dump) I though would be a good idea to give some more details. And just to jump start on the topic, here's something I learnt a while ago after wasting a few hours typing commands and looking at inconsistent results... debugging with the wrong symbols could be much worse than debugging with no symbols at all.

What are symbols?

You can think of symbol files basically as small databases, files which contain source line information, data types, variables, functions and everything else needed to provide names for all lines of code, instead of hexadecimal addresses. They usually have .pdb (or .dbg) extension, and are matched with the actual executable code using an internal timestamp, so it's very important to generate symbols every time you build the application, also for release builds. If you'll ever have a problem with your live application and you'll need to debug it, and if you'll not have the matching symbols (matching means the symbols obtained from the same exact build of the dlls you put in production) you could be in troubles... Building once again the application to create the symbol files, even without changing your code does not really help because the timestamp will not match, and WinDbg will complain about missing symbols anyway...

Note that in Visual Studio if you set "Release" in the "Standard" toolbar, the "Generate debugging information" checkbox will automatically be cleared, so remember to go to the Project properties and flag it again before rebuilding the project.

Since in ASP.NET 2.0 we have a new default page architecture, there is no real need to create the .pdb files unless you're using the "Web Application Project" template which comes with the Service Pack 1 for Visual Studio 2005, in which case you can find it from the project properties, "Compile" tab.

The same applies for Visual Studio 2008.

How can we use symbols?

When you open a dump within WinDbg and you type in a command for example to inspect the call stack, the debugger will start looking for matching symbols to show you an output as detailed as possible: but how does it decide where to look for those symbols? It will use the path(s) specified in the "Symbol Search Path" dialog you can find under the File menu (CTRL+S is the keyboard shortcut.

Here you can specify the symbol servers where WinDbg can download the symbols from, and of course you can use has many server more than one server at a time; WinDbg will simply access those servers in the same order you put in the search path, and it goes through the end of the list until it finds a match.

Microsoft has a public symbol server accessible through the Internet which stores public .pdb files for almost all of our products: http://msdl.microsoft.com/download/symbols.

I work a lot with WinDbg and memory dumps in my daily job, also with my laptop when not connected to the Internet or corporate network, so I need to be able to debug while offline and in any case I don't want to waste time waiting for the tool to download the same symbols over and over again, through one dump to the other... for this reason it's also possible to create a local symbol cache (also referred as downstream store) on your hard disk. When looking for symbols, WinDbg will first of all check your local cache and if the match is found there no additional check is done against the other symbol servers, while if the match is not found WinDbg goes on as usual until it finds the right match; in this case it downloads the .pdb and stores it in your local symbol cache, so that next time it will be readily available for you.

The symbol path is a string composed of multiple directory paths, separated by semicolons. For each directory in the symbol path, the debugger will look in three directories: for instance, if the symbol path includes the directory c:\MyDir, and the debugger is looking for symbol information for a dll, the debugger will first look in c:\MyDir\symbols\dll, then in c:\MyDir\dll, and finally in c:\MyDir. It will repeat this for each directory in the symbol path. Finally, it will look in the current directory, and then the current directory with \dll appended to it. (The debugger will append dll, exe, or sys, depending on what binaries it is debugging.)

Here is a sample symbol path:

SRV*c:\symbols*\\internalshare\symbols*http://msdl.microsoft.com/download/symbols

The above is actually the symbol path (a bit simplified) I use on my machines: as you can see I have a local cache in C:\Symbols; if I've never downloaded a particular symbol before, WinDbg does to an internal share were we have full symbols, and if still unsuccessful I finally give a try to the public Microsoft symbol server on the Internet. If you include two asterisks in a row where a downstream store would normally be specified, then the default downstream store is used. This store will be located in the sym subdirectory of the home directory. The home directory defaults to the debugger installation directory; this can be changed by using the !homedir extension. If the DownstreamStore parameter is omitted and no extra asterisk is included (i,e. if you use srv with exactly one asterisk or symsrv with exactly two asterisks) then no downstream store will be created and the debugger will load all symbol files directly from the server, without caching them locally. Note that If you are accessing symbols from an HTTP or HTTPS site, or if the symbol store uses compressed files, a downstream store is always used. If no downstream store is specified, one will be created in the sym subdirectory of the home directory.

The symbol server does not have to be the only entry in the symbol path. If the symbol path consists of multiple entries, the debugger checks each entry for the needed symbols; moreover the symbol path can contain several directories or symbol servers, separated by semicolons. This allows you to locate symbols from multiple locations (or even multiple symbol servers). If a binary has a mismatched symbol file, the debugger cannot locate it using the symbol server because it checks only for the exact parameters. However, the debugger may find a mismatched symbol file with the correct name, using the traditional symbol path, and successfully load it; in this case it's important to know if our symbols matches (see the next topic).

You can set the symbol path in advance once for all within WinDbg: open an empty instance, press CTRL+S, type in the path, clock "Ok" on the dialog and close WinDbg, accepting to save the workspace if prompted to do so (next time you'll open WinDbg the value will still be there). Or you can use the .sympath command within WinDbg with a dump open.

Another option is to set the system wide variable _NT_SYMBOL_PATH (the syntax is still the same), used by debuggers like WinDbg or also from adplus directly when it captures the dump.

The same principle applies to the Visual Studio debugger (also have a look at the article http://support.microsoft.com/kb/311503/en-us):

How can I check if my symbols matches?

Looking at a call stack sometimes it's clear you're having a problem with unmatched symbols because WinDbg tells you something like:

 1: ChildEBP RetAddr 

 2: 0012f6dc 7c59a2d1 NTDLL!NtDelayExecution(void)+0xb

 3: 0012f6fc 7c59a29c KERNEL32!SleepEx(unsigned long dwMilliseconds = 0xfa, int bAlertable = 0)+0x32

 4: *** ERROR: Symbol file could not be found. Defaulted to export symbols for aspnet_wp.exe - 

 5: 0012f708 00442f5f KERNEL32!Sleep(unsigned long dwMilliseconds = 0x444220)+0xb

 6: WARNING: Stack unwind information not available. Following frames may be wrong.

 7: 0012ff60 00444220 aspnet_wp+0x2f5f

 8: 0012ffc0 7c5989a5 aspnet_wp!PMGetStartTimeStamp+0x676

 9: 0012fff0 00000000 KERNEL32!BaseProcessStart(<function> * lpStartAddress = 0x004440dd)+0x3d

Unfortunately could happen to not be so lucky, and you'll find yourself wondering if the stack you are looking at is genuine or there are some small (or maybe even not so small) inconsistencies which may lead you down to a completely wrong path. In such cases, you can first of all use the lm command to find which .pdb files have been loaded:

 1: kernel32 (pdb symbols) .sympath SRV\kernel32.pdb\CE65FAF896A046629C9EC86F626344302\kernel32.pdb

 2: ntdll (pdb symbols) .sympath SRV\ntdll.pdb\36515FB5D04345E491F672FA2E2878C02\ntdll.pdb

 3: shell32 (deferred)

 4: user32 (deferred)

As you can see in the example above, two symbols were loaded (for kernel32.dll and ntdll.dll), while shell32.dll and user32.dll were not part of the stack analyzed, so WinDbg has not loaded yet (deferred) their symbols. A bad match will look like the following:

 1: ntdll M (pdb symbols) .sympath SRV\ntdll.pdb\36515FB5D04345E491F672FA2E2878C02\ntdll.pdb

Notice the "M" highlighted in red (could also be a "#" pound sign)? That stands or "mismatch", and indicates there is a problem with that particular module (search for "Symbol Status Abbreviations" in WinDbg help for further details). Alternatively you can use the !sym noisy and .reload command to reload symbols verbosely to have a detailed output. Look for "Symbols files and paths - Overivew" in WinDbg help for more details.

Trick: you have the right symbol, but WinDbg does not matches it anyway...

I'm not sure why this happens, and especially why I had this problem only with ntdll.dll (and its .pdb): I was not able to get the proper stack even with a matching symbol (and I checked more than once to be really sure)... until I got the idea to delete the ntdll.pdb folder in my local cache (if you have a dump open you must first unload the symbol from WinDbg or the file will be locked: use the .reload /u <module_name> command), then run a .reload /f <module_name> (/f forces immediate symbol load) and let WinDbg to download it again... this usually does the trick and I finally get the correct stack.

Debugging without symbols?

It's not impossible, but it's harder than debugging with matching symbols; the main difference is that you'll not be able to see method names, variable names etc... and generally speaking the stack will be less easily readable. To give you a quick example, here is an excerpt of the stack of a very simple application I wrote for test (it has just a button which sets the text of a label to the DateTime.Current.ToString()):

without symbols:

 1: 5 Id: 10d4.1204 Suspend: 1 Teb: 7ffd7000 Unfrozen

 2: ldEBP RetAddr 

 3: NING: Frame IP not in any known module. Following frames may be wrong.

 4: 9f524 71a6b7f8 0x7c90eb94

 5: 9fa0c 03490657 0x71a6b7f8

 6: WARNING: Unable to verify checksum for System.dll

 7: ERROR: Module load completed but symbols could not be loaded for System.dll

 8: 9fa40 7a603543 CLRStub[StubLinkStub]@3490657(<Win32 error 318>)

 9: a8240 032908ff System!System.Net.Sockets.Socket.Accept(<HRESULT 0x80004001>)+0xc7

 10: ERROR: Module load completed but symbols could not be loaded for WebDev.WebHost.dll

 11: 9fab0 7940a67a WebDev_WebHost!Microsoft.VisualStudio.WebHost.Server.OnStart(<HRESULT 0x80004001>)+0x27

 12: WARNING: Unable to verify checksum for mscorlib.dll

 13: ERROR: Module load completed but symbols could not be loaded for mscorlib.dll

 14: bd1b4 7937d2bd mscorlib!System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(<HRESULT 0x80004001>)+0x1a

 15: bd1b4 7940a7d8 mscorlib!System.Threading.ExecutionContext.Run(<HRESULT 0x80004001>)+0x81

 16: 9fae0 7940a75c mscorlib!System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(<HRESULT 0x80004001>)+0x44

 17: 32010 79e79dd3 mscorlib!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(<HRESULT 0x80004001>)+0x60

 18: 9fb04 79e79d57 0x79e79dd3

 19: 9fb84 79f71cba 0x79e79d57

 20: 9fba4 79f71c64 0x79f71cba

 21: 9fc08 79f71cf3 0x79f71c64

 22: 9fc3c 7a0b0896 0x79f71cf3

 23: 9fc9c 79f7ba4f 0x7a0b0896

 24: 9fcb0 79f7b9eb 0x79f7ba4f

 25: 9fd44 79f7b90c 0x79f7b9eb

 26: 9fd80 79ef9887 0x79f7b90c

 27: 9fda8 79ef985e 0x79ef9887

 28: 9fdc0 7a0a32da 0x79ef985e

 29: 9fe28 79ef938f 0x7a0a32da

 30: 9fe94 79f7be67 0x79ef938f

 31: 9ffb4 7c80b683 0x79f7be67

 32: 9ffec 00000000 0x7c80b683

with matching symbols:

 1: 5 Id: 10d4.1204 Suspend: 1 Teb: 7ffd7000 Unfrozen

 2: ldEBP RetAddr 

 3: 9f4e4 7c90e9c0 ntdll!KiFastSystemCallRet

 4: 9f4e8 71a54033 ntdll!ZwWaitForSingleObject+0xc

 5: 9f524 71a6b7f8 mswsock!SockWaitForSingleObject+0x1a0

 6: 9f9bc 71ac0e2e mswsock!WSPAccept+0x21f

 7: 9f9f0 71ac103f ws2_32!WSAAccept+0x85

 8: 9fa0c 03490657 ws2_32!accept+0x17

 9: WARNING: Unable to verify checksum for System.ni.dll

 10: 9fa40 7a603543 CLRStub[StubLinkStub]@3490657(<Win32 error 318>)

 11: a8240 032908ff System_ni!System.Net.Sockets.Socket.Accept(<HRESULT 0x80004001>)+0xc7

 12: 9fab0 7940a67a WebDev_WebHost!Microsoft.VisualStudio.WebHost.Server.OnStart(<HRESULT 0x80004001>)+0x27

 13: WARNING: Unable to verify checksum for mscorlib.ni.dll

 14: bd1b4 7937d2bd mscorlib_ni!System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(<HRESULT 0x80004001>)+0x1a

 15: bd1b4 7940a7d8 mscorlib_ni!System.Threading.ExecutionContext.Run(<HRESULT 0x80004001>)+0x81

 16: 9fae0 7940a75c mscorlib_ni!System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(<HRESULT 0x80004001>)+0x44

 17: 32010 79e79dd3 mscorlib_ni!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(<HRESULT 0x80004001>)+0x60

 18: 9fb04 79e79d57 mscorwks!CallDescrWorker+0x33

 19: 9fb84 79f71cba mscorwks!CallDescrWorkerWithHandler+0xa3

 20: 9fba4 79f71c64 mscorwks!DispatchCallBody+0x1e

 21: 9fc08 79f71cf3 mscorwks!DispatchCallDebuggerWrapper+0x3d

 22: 9fc3c 7a0b0896 mscorwks!DispatchCallNoEH+0x51

 23: 9fc9c 79f7ba4f mscorwks!QueueUserWorkItemManagedCallback+0x6c

 24: 9fcb0 79f7b9eb mscorwks!Thread::DoADCallBack+0x32a

 25: 9fd44 79f7b90c mscorwks!Thread::ShouldChangeAbortToUnload+0xe3

 26: 9fd80 79ef9887 mscorwks!Thread::ShouldChangeAbortToUnload+0x30a

 27: 9fda8 79ef985e mscorwks!Thread::ShouldChangeAbortToUnload+0x33e

 28: 9fdc0 7a0a32da mscorwks!ManagedThreadBase::ThreadPool+0x13

 29: 9fe28 79ef938f mscorwks!ManagedPerAppDomainTPCount::DispatchWorkItem+0xdb

 30: 9fe3c 79ef926b mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0xaf

 31: 9fe94 79f7be67 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x223

 32: 9ffb4 7c80b683 mscorwks!Thread::intermediateThreadProc+0x49

 33: 9ffec 00000000 kernel32!BaseThreadStart+0x37

The difference is quite obvious... The WinDbg help file also gives a few hints:

1. To figure out what the addresses mean, you'll need a computer which matches the one with the error. It should have the same platform (x86, Intel Itanium, or x64) and be loaded with the same version of Windows
2. When you have the computer configured, copy the user-mode symbols and the binaries you want to debug onto the new machine
3. Start CDB or WinDbg on the symbol-less machine
4. If you don't know which application failed on the symbol-less machine, issue an | (Process Status) command. If that doesn't give you a name, break into KD on the symbol-less machine and do a !process 0 0, looking for the process ID given by the CDB command
5. When you have the two debuggers set up — one with symbols which hasn't hit the error, and one which has hit the error but is without symbols — issue a k (Display Stack Backtrace) command on the symbol-less machine
6. On the machine with symbols, issue a u (Unassemble) command for each address given on the symbol-less stack. This will give you the stack trace for the error on the symbol-less machine
7. By looking at a stack trace you can see the module and function names involved in the call

I got symbols from my customer: what should I do now?

The easiest thing you could do is use symstore.exe (you'll find it in WinDbg/adplus installation folder) with a command like the following:

symstore add /f c:\temp\SymbolsTest\Bin\*.pdb /s c:\symbols /t "Symbols Test"

Let's have a quick look at the syntax:

• The "add" keyword is quite self explanatory
• /f tells symstore which is the file you want to add; note you can use wildcards, so you can add multiple files at once
• /s is the path to your symbols store (this will most likely be your local cache, or your shared symbol server)
• /t a required description for the symbol to store

Symstore will create a structure similar to the following:

Also note the "0000Admin" folder created by symstore, which contains one file for each transaction (every "add" or "delete" operation is recorded as a transaction), as well as the logs server.txt and history.txt; the former contains a list of all transactions currently on the server, while the latter contains a chronological history of all transactions run on the machine. For further information you can see the "Using SymStore" topic in WinDbg help (debugger.chm).

Conclusion

While it is possible to debug without symbols (this is true especially for managed code), remember that your life could be much easier if you (and your customers) will take care of your symbols and will generate them every time the application will be rebuilt, also in release mode.

I deliberately simplified the argument (I just wanted to share what I learnt in my daily job and give some quick tips to get started), much more could be said and if you're interested I encourage you to read the "Symbols" section in the WinDbg help, or search the Internet where you'll find good blog posts/articles on this subject

Carlo