Esoteric : Production debugging

Quick Tip: vfbasics!_AVRF_EXCEPTION_LOG_ENTRY symbol not resolved?

arvindsh — Fri, 21 Aug 2009 13:17:31 GMT

I was debugging some issues with the help of Application Verifier and WinDbg. Since I was onsite I did not have any access to Microsoft’s private symbol servers, so I was using the public symbol server (http://msdl.microsoft.com/download/symbols). On executing the !avrf extension command in WinDbg I was presented with the following error message in WinDbg:

***    Your debugger is not using the correct symbols                 ***
***                                                                   ***
***    In order for this command to work properly, your symbol path   ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information. Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: vfbasics!_AVRF_EXCEPTION_LOG_ENTRY                ***

It turns out that my WinDbg symbol path was as follows, and due to it pointing just to the public symbol server it was loading public symbols for vfbasics.dll:

0:001> .sympath
Symbol search path is: SRV*c:\localsymbols*http://msdl.microsoft.com/download/symbols
Expanded Symbol search path is: srv*c:\localsymbols*http://msdl.microsoft.com/download/symbols
0:001> lml
start             end                 module name
00000000`76dc0000 00000000`76f68000   ntdll      (pdb symbols)          c:\localsymbols\ntdll.pdb\FDAD9EE7D6E44F4F9672ECB401A802192\ntdll.pdb
000007fe`f0e50000 000007fe`f0ebe000   verifier   (pdb symbols)          c:\localsymbols\verifier.pdb\43FCE2D63C4544F9B1C67110EB3406951\verifier.pdb
000007fe`f1660000 000007fe`f1693000   vrfcore    (pdb symbols)          c:\localsymbols\vrfcore.pdb\751D23CCD6504794AF2F18C1E547FE371\vrfcore.pdb
000007fe`f28e0000 000007fe`f292a000   vfbasics   (pdb symbols)          c:\localsymbols\vfbasics.pdb\01ABCDFEFF9F4602A7F055801457A7D61\vfbasics.pdb

To resolve the issue, I explicitly pre-pended the path to private symbols for vfbasics (which is c:\windows\system32 or in general %WINDIR%\System32:

.sympath c:\windows\system32*SRV*c:\localsymbols*http://msdl.microsoft.com/download/symbols

.reload

ld vfbasics

0:001> lml
start             end                 module name
00000000`76dc0000 00000000`76f68000   ntdll      (export symbols)       C:\Windows\SYSTEM32\ntdll.dll
000007fe`f28e0000 000007fe`f292a000   vfbasics   (private pdb symbols) C:\Windows\SYSTEM32\vfbasics.pdb

Then !avrf works just fine!

If you liked this post, please do rate it and try to leave some comments if you can!

Debugging Toolbox

arvindsh — Fri, 30 Jan 2009 22:51:01 GMT

This one is a quickie for an easy reference to most commonly used debugging tools and links. I hope you find it useful, and kindly indicate your feedback on this page by using the comments section or by rating the post!

Debugging Toolbox

Tool	Key Usage Scenarios	Download location
WinDbg	Interactive production debugging Dump analysis Local kernel debugging	http://www.microsoft.com/whdc/devtools/debugging/default.mspx
CDB	Scriptable, low overhead capture of dumps Low overhead interactive debugging
ADPlus	Scripted automation for capturing dumps
DebugDiag	Can setup rules for capturing dumps in an unattended, logged-off scenario Automated analysis of basic crash, hang and leak scenarios	http://www.iis.net
SysInternals Process Monitor	Interactive process and thread level details Handle usage	http://www.sysinternals.net
TLIST	Listing services Listing process tree Listing which processes have loaded a module	http://www.microsoft.com/whdc/devtools/debugging/default.mspx
Performance Monitor	Interactive display of performance data on the system (System Monitor) Logged-off unattended capturing of performance logs	Shipped with the OS
Application Verifier	Enable various checks to trap deep rooted issues earlier such as Orphaned critical sections Heap corruption Unsafe API usage Simulate low memory conditions	http://msdn.microsoft.com/en-us/library/aa480483.aspx http://www.microsoft.com/downloads/details.aspx?FamilyID=C4A25AB9-649D-4A1B-B4A7-C9D8B095DF18&displaylang=en
PREFast	Static source code analysis to detect potential buffer / stack overruns etc.	http://www.microsoft.com/whdc/devtools/tools/prefast.mspx
XPerf / XPerfInfo	Profiler / tracing of user mode applications	http://msdn.microsoft.com/en-us/library/cc305187.aspx http://msdn.microsoft.com/en-us/performance/cc825801.aspx http://blogs.msdn.com/seema/archive/2008/10/08/xperf-a-cpu-sampler-for-silverlight.aspx
UserDump	Automated capture of dumps	http://www.microsoft.com/downloads/details.aspx?FamilyID=e089ca41-6a87-40c8-bf69-28ac08570b7e&DisplayLang=en
LeakDiag	Leak diagnostics	ftp://ftp.microsoft.com/PSS/Tools/Developer%20Support%20Tools/LeakDiag

Native code debugging sites and blogs

http://www.advancedwindowsdebugging.com

http://blogs.msdn.com/ntdebugging

http://www.osronline.com

http://www.alex-ionscu.com

http://www.dumpanalysis.org

http://www.nynaeve.net/

http://blogs.thinktecture.com/ingo/archive/2006/08/05/414674.aspx

.NET articles:

http://msdn2.microsoft.com/en-us/library/ms954594.aspx

http://msdn.microsoft.com/msdnmag/issues/03/06/Bugslayer

http://msdn.microsoft.com/msdnmag/issues/05/03/Bugslayer

http://support.microsoft.com/kb/892277

http://msdn.microsoft.com/msdnmag/issues/05/07/Debugging/

http://msdn.microsoft.com/msdnmag/issues/06/11/CLRInsideOut/default.aspx

.NET debugging blogs:

http://blogs.msdn.com/yunjin/

http://blogs.msdn.com/tess/

http://blogs.msdn.com/jmstall/

http://blogs.msdn.com/mvstanton/

http://blogs.msdn.com/cbrumme/

http://blogs.msdn.com/maoni/

http://blogs.msdn.com/toddca/

http://blogs.msdn.com/suzcook/

http://www.debugtricks.com

Symbols and symbol files

arvindsh — Wed, 30 Aug 2006 07:44:00 GMT

Some of us are unaware of the relevance of proper symbols files. The symbol file is a deliverable just like the binary. Though it is not required for running the application, it is very important to debug issues which arise in production.

What are symbols and symbol files?

Symbols contain the mapping between the compiler-generated machine code and your source code. They help the debugger to "understand" the addresses of functions, parameters and variables and map them back to source code references. Without a proper symbol file, the debugger may present disassembly output which looks like this:

call YourApp+0x11c0d (00411c0d)

With proper symbol resolution, the debugger can correctly interpret this machine code as the equivalent function name:

call YourApp!CSampleDlg::OnBtnClicked

In the Visual Studio family of products, we use the PDB (Program Database) format for encoding the symbolic information into a file. Given below are the steps to generate the symbol files for various product versions. Please ensure that the PDB files generated are of the same code base as the service EXE/DLLs that are actually used in the testing. Everytime your code is rebuilt, the PDB is re-generated and has to be maintained in sync with your code base and binaries.

Private vs. Public symbols

Public symbol files are what you get from Microsoft for the Windows OS and other products. You can obtain public symbols for Microsoft products either by referencing the public symbol server at http://msdl.microsoft.com/download/symbols, or by downloading appropriate symbol packages from http://www.microsoft.com/whdc/devtools/debugging/default.mspx. Public symbols normally just contain:

Mapping from addresses to function names
Frame pointer omission (FPO) records

Private symbol files are the default kind of files generated by the Visual Studio linkers. These files contain the following details in addition to what the public symbols contain:

Line numbers and source file information
Parameter names and data types
Local variable names and data types

Normally you would want private symbol files for any detailed debugging of your own application. However, it may not be a very good idea to share your private PDB files with your end customers, since some of the information in the private PDB can be viewed as intellectual property.

To generate stripped / public PDB files with Visual C++, use the information in http://msdn2.microsoft.com/en-US/library/y87kw2fd.aspx for steps.

How to generate symbol files for Visual C++.NET 7/7.1/8.0

In these versions of VC++, a symbol file should automatically be created if you are compiling a debug build. If you are compiling a release build, then you will need to check the steps below to ensure that linker will create symbols. Please perform the symbol generation steps for each and every module in your application. If it loads other DLLs then these steps need to be done for those DLLs also.

Open the source code for your project.
Open the properties page for the project from Solution Explorer
View the Configuration Properties->Linker->Debugging node
Make sure "Generate Debug Info" is set to YES
Also check that the "Generate Program Database File" is set to a non-blank path
Rebuild the project to generate the PDB file

How to generate symbol files for Visual C#.NET 7/7.1/8.0

In VC#, a symbol file should automatically be created if you are compiling a debug build. If you are compiling a release build, the steps are given below.

Open the source code for your project.
Open the properties page for the project from Solution Explorer
View Configuration Properties->Build node
Make sure "Generate Debugging Information" is set to True.
Rebuild the project to generate the PDB file

How to generate symbol files for Visual Basic.NET 7/7.1/8.0

In VB.NET, a symbol file should automatically be created if you are compiling a debug build. If you are compiling a release build, the steps are given below.

Open the source code for your project.
Open the properties page for the project from Solution Explorer
View Configuration Properties->Build node
Make sure "Generate Debugging Information" is checked.
Rebuild the project to generate the PDB file

How to generate symbol files for Visual C++ 6.0

Important note: if you are still using VC++ 6.0, be aware that Microsoft Support will not be in a position to support you on issues concerning the product. That is because the end of support lifecycle has been reached for this version of VC++. For more information, review the details at http://support.microsoft.com/lifecycle and plan to upgrade to higher versions of VC++.NET. The information below is provided for reference only, and should not be interpreted as a sign of support!

In VC++ 6.0, a symbol file should automatically be created if you are compiling a debug build. If you are compiling a release build, then you will need to follow these steps to create symbols. Please perform the symbol generation steps for each and every module in your application. If it loads other DLLs then these steps need to be done for those DLLs also.

Open the source code for your project.
Select Settings from the Project menu.
In the Link tab, select General from the Category drop-down.
Check Generate Debug Info
In the C/C++ tab, select General from the Category drop-down.
Select Program Database from the Debug Info drop-down.
Link with /RELEASE to get a non-zero checksum.
Go to the Category drop-down & select Listing Files.
In the Listing File drop-down, select 'Assembly, Machine Code, and Source'.
Rebuild your project.

How to generate symbol files for Visual Basic 6.0

In VB6, symbols are not created by default. To create symbols for your VB6 applications, follow these steps:

Open the source code for your project.
Select "<projectname> Properties" from the Project menu.
In the Compile tab, check Create Symbolic Debug Info and select No Optimizations
Recompile. This will create a .PDB file (the symbols) in the same directory as your .DLL / .EXE / .OCX file.
Note: This does *not* break binary compatibility.