If broken it is, fix it you should

Using the powers of the debugger to solve the problems of the world - and a bag of chips    by Tess Ferrandez, ASP.NET Escalation Engineer (Microsoft)

Capturing memory dumps for 32-bit processes on an x64 machine

Capturing memory dumps for 32-bit processes on an x64 machine

Rate This
  • Comments 17

This is an issue that I often get questions around and we often have cases where we have to re-capture memory dumps because the memory dumps were captured the “wrong” way.

The short story is: If you are executing a 32-bit process on a 64-bit machine, (which is the default case for IIS on x64 machines for example) you need to capture the dump with a tool that allows you to create 32-bit dumps.

How do you know if your process is 32-bit?

If you are on a 64-bit machine, you can check task manager to see what architecture your process is using.

image

Processes with *32 are 32-bit and the rest are 64-bit so in the example above we can see that for example w3wp.exe is executing 32-bit code.

Why is it important to capture them with the right tools?

If you capture a dump with a tool that captures 64-bit dumps you will still get a memory dump, but you will get a memory dump of the syswow64 which means that a lot of extensions like sos and psscor2 won’t be able to read the data.  Some things may still work but it is very limited and you might get errors or erroneous call stacks etc.

Typical stuff that happens when you try to read a 64-bit memory dump of a 32-bit process is:

1. You may see warnings like

  WARNING: wkscli overlaps srvcli
  ..............WARNING: wship6 overlaps dnsapi
  .WARNING: IPHLPAPI overlaps dsrole
  ...WARNING: FWPUCLNT overlaps rasadhlp
  WARNING: FWPUCLNT overlaps dnsapi
  .....WARNING: compstat overlaps iisres

2.  The stacks show wow64cpu methods

0:000> kp 
   Child-SP          RetAddr           Call Site 
   00000000`000ce728 00000000`73a22bcd wow64cpu!CpupSyscallStub+0x9 
   00000000`000ce730 00000000`73a9d07e wow64cpu!Thunk0ArgReloadState+0x1a 
   00000000`000ce7f0 00000000`73a9c549 wow64!RunCpuSimulation+0xa 
   00000000`000ce840 00000000`76d684c8 wow64!Wow64LdrpInitialize+0x429 
   00000000`000ced90 00000000`76d67623 ntdll!LdrpInitializeProcess+0x17e2 
   00000000`000cf290 00000000`76d5308e ntdll! ?? ::FNODOBFM::`string'+0x2bea0 
   00000000`000cf300 00000000`00000000 ntdll!LdrInitializeThunk+0xe

3. You see that all addresses are 64 bit addresses, (i.e. 00000000`76d5308e rather than 76d5308e), even though it is a 32-bit process.

4. You get errors like the below when trying to run sos commands.  
(Note: You can get these errors if you don’t have symbols properly set up as well, so this is not the only reason for getting these errors)

0:000> !eeheap -gc 
Failed to load data access DLL, 0x80004005 
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)             2) the file mscordacwks.dll that matches your version of clr.dll is                 in the version directory             3) or, if you are debugging a dump file, verify that the file                 mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.             4) you are debugging on the same architecture as the dump file.                 For example, an IA64 dump file must be debugged on an IA64                 machine. 

You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll.  .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.

If you are debugging a minidump, you need to make sure that your executable
path is pointing to clr.dll as well.

What tools should you use?

The best way to capture a process dump in this scenario is to use 32-bit versions of tools like Debug Diag, or adplus+cdb (32 bit debugging tools for windows).  You can install the 32-bit versions of these tools on a 64-bit system.   In the next version of Debug Diag you will be able to capture 32-bit dumps with the 64-bit version of Debug Diag.

I have mentioned before that you can capture memory dumps with task manager on Vista+, but if you do this on a 64-bit machine you will get a 64-bit dump.  You can however use the 32-bit task manager, located in C:\Windows\SysWOW64\taskmgr.exe to get 32-bit dumps.  If you want to verify that you are running the 32-bit version of task manager, you can check that taskmgr.exe is listed with *32 in task manager itself.

Edit:  Sorry Mark and David, I of course didn't mean to leave out the nice and very lightweight procdump tool from SysInternals which will capture 32-bit dumps of 32-bit processes even on 64-bit Windows.  Thanks for reminding me:)
For some reason I thought that I had written about this tool in an earlier post but a quick search reminded me that this is still in my blog-todo bin. 

For all other dump capturing tools, I would suggest that you read the documentation to see if there is a 32-bit version (like for procmon where you can pass in the /run32 parameter) or if they will capture 32-bit dumps of 32-bit processes.

Is it important to read the dump with the right “bit” debugger?

Yes, to read 32-bit dumps you should use 32-bit windbg (either on a 32-bit or 64-bit OS), and for 64-bit dumps, the machine that you debug the dumps on, needs to be a 64-bit machine, running 64-bit windbg.

Remember that you should also use the 64-bit versions of sos, psscor2, sosex and any other extensions that you might use, if you are debugging a 64-bit dump.

 

Have a good one,
/Tess

  • A way who also worked for me for an w3wp*32 Process on a 64-bit Machine was capturing a dump with sysinternals procdump utility (a single file, doesn't require installation, can capture explicit 32-bit or 64-bit dumps on 64-bit Machines), transferring dump to a 32-bit Dev-Machine with 32-bit WinDgb and before using SOS i had to issue ".exepath+ C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\" as described here:

    voneinem-windbg.blogspot.com/.../failed-to-load-data-access-dll.html

    Worked perfect!

  • Tess,

    In my experience, w3wp defaults to 64 bit on 64 bit windows.  32 bit is enabled when I change the app pool to 'enable 32 bit applications'

    John

  • Tess - am really surprised (Russinovich was saddened) that you don't mention Procdump in your article - that surely would be the preferred tool since it is so lightweight (no install like the tools you suggestion).  So just curious - why didn't you recommend it?

    Also, technical detail: there is no such thing as a 32-bit process on 64-bit Windows - a Wow64 process is a full 64-bit process who's threads start out in 64-bit mode but then switch to 32-bit mode to execute 32-bit code.  And when the 32-bit code calls system calls, a switch is made back to 64-bit mode to call 64-bit OS services.  You say above in #3 "You see that all addresses are 64 bit addresses, (i.e. 00000000`76d5308e rather than 76d5308e), even though it is a 32-bit process" - in fact, this is  proof that it is a 64-bit thread (Windbg can show the 32-bit stack as well).

  • And procdump has a significant number of other features that make it much more flexible and applicable to more scenarios....

    -Mark (with a heavy heart)

  • Well if we're going to be technical, there's also no such thing as a "running" process, as Windows schedules threads, not processes. So you're really "executing a Wow64 process" J

  • Mark, David,

    I was actually thinking of adding it to the list before along with a few other dump tools, but the list got a bit long, so I just included debug diag and adplus and left the others under (other dump creation tools).

    The reason I singled debug diag and adplus out was because of the types of issues i normally deal with where you often need to do leaktracking or dump on specific exceptions (handled).  For gathering dumps on "hangs", or straight snapshots for memory analysis we usually use taskmgr (because it is already there).  Having said that, I personally like procdump a lot and use it very often, and I do agree that it does have a lot of neat features like gathering dumps on memory/CPU threasholds and based on perfcounter info.  

    As I wrote in the post (in the edit) I thought I had written a post about it already but just realized it is still in my todo bin, but I think it does merit it's own post.

    On a separate note, since I love the nitpicking:)  perhaps you should change the help then for procdump technet.microsoft.com/.../dd996900.aspx from "By default Procdump will capture a 32-bit dump of a 32-bit process when running on 64-bit Windows." to ... a 32-bit dump of a 64-bit process running 32-bit code:) ... sorry couldn't resist, thanks for the clarification btw...

    aionescu, thanks:)  changed that, i'll blame it on my Swenglish (Swedish - English) :)

  • It's fairly easy to see the 32-bit view of the world if you start seeing wow64! show up in the call trace, just run:

    !wow64exts.sw

    I'm 85% sure that this will also let SOS work correctly - no need to worry about being able to decode an "Incorrect Arch" dump!

  • Paul,  

    Thanks for pointing out the !wow64exts.sw command, it's definitely very useful in this case.

    Unfortunately it doesn't fix the whole situation though... it does let you use some commands for sos like !clrstack and other stack related stuff, but for anything that requires sos to look at the heaps, like !dumpheap -stat, !threads, etc. etc. it doesnt work unfortunately.  

    I am not sure what the sw command actually does internally but I have a feeling that it does something similar to changing process contexts with .process in kernel dumps which has similar effects on sos.  Perhaps someone else can shed some more light, but either way, to get the full sos functionality you need to get 32-bit dumps...

  • Hi Tess,

    Is there way to get the information, which calls into the dbgHelp.dll have to be done to get a 'correct' 32 bit process dump out of a 64 bit executable.

    The current tools are pretty nice, but we can't use them.

    We are having a lot of processes to monitor automatically during development.

    Therefore we have an own tool, which attaches to every process of interrest within a single thread and creates .dmp files

    depending on the type/address of an exception.

    For getting the correct information out of the single processes, the tool has to be run in 32 bit on 32 bit OSes and as a 64 bit one on 64 bit OSes, respetively.

    But, as you mentioned above, how to call MiniDumpWriteDump out of a 64 bit process for a 32 bit target?

    thanks in advance

    andy

  • Hi Tess,

    The above didn't work for me, got "Operation could not be completed. Only part of the memory was written". We have a x64 server and 32 bit development machines. Our memory of the process (ASP.NET) is hitting 2.2gb! I'm sure you get asked this lots but could have you have a 10min glance over it and see where you think we should be looking.

  • Procdump is the nice and very lightweight tool to capture 32-bit dumps of 32-bit processes on 64-bit Windows. However, based on the license of ProcDump, we could not use it within our product.

    Any suggestion of capturing 32-bit dumps on 64-bit Windows in the program.

  • @Leo:  have a look at channel9.msdn.com/.../Devdays084

    esp. 1hr 09 min

  • @Mark Russinovich  I have a feature request for Process Explorer.

    Version 15 makes a 64-bit dump of 32-bit processes...  Please help our testers.

  • You are my hero... I wanna cry :( thanks

  • I second the request of Gregory Chernis. The latest Process Explorer 64 bit, V 15.13 still dumps 64 bit of 32 bit processes.

Page 1 of 2 (17 items) 12
Leave a Comment
  • Please add 8 and 1 and type the answer here:
  • Post