If broken it is, fix it you should

Using the powers of the debugger to solve the problems of the world - and a bag of chips    by Tess Ferrandez, ASP.NET Escalation Engineer (Microsoft)

.NET Debugging Demos Lab5: Crash - Review

.NET Debugging Demos Lab5: Crash - Review

  • Comments 26

 

Since I already posted a challenge for this lab earlier I didn't want to wait too long with publishing the review... 

Previous demos and setup instructions

If you are new to the debugging labs, here you can find information on how to set up the labs as well as links to the previous labs in the series.

Information and setup instructions
Lab 1: Hang
Lab 1: Hang - review
Lab 2: Crash
Lab 2: Crash - review
Lab 3: Memory
Lab 3: Memory - review
Lab 4: High CPU hang
Lab 4: High CPU hang - review

Reproducing the issue:

1. Recycle IIS (IISReset)

2. Browse to http://localhost/BuggyBits/CompanyInformation.aspx 

3. Type some message and click the Send button 

Note: If just-in-time debugging is enabled a message box might pop up because the w3wp.exe process crashed.  If this comes up, please ignore it.

Q: What do you see in the browser?

A: A "Internet Explrer cannot display the webpage" page indicating that something is wrong on the server.

Explore the eventlogs

1. Open the eventlogs for system an application

Q: What events do you notice relating to the crash? (note: you may see different events on different operating systems)

A:
Application

Event Type: Error
Event Source: .NET Runtime 2.0 Error Reporting
Event Category: None
Event ID: 1000
Date: 2008-03-04
Time: 12:57:33
User: N/A
Computer: MYMACHINE
Description:
Faulting application w3wp.exe, version 6.0.3790.3959, stamp 45d6968e, faulting module kernel32.dll, version 5.2.3790.4062, stamp 4626467c, debug? 0, fault address 0x00022366.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
...

System

Event Type: Warning
Event Source: W3SVC
Event Category: None
Event ID: 1009
Date: 2008-03-04
Time: 12:57:19
User: N/A
Computer: MYMACHINE
Description:
A process serving application pool 'DefaultAppPool' terminated unexpectedly. The process id was '5764'. The process exit code was '0x800703e9'.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Event Type: Warning
Event Source: W3SVC
Event Category: None
Event ID: 1011
Date: 2008-03-04
Time: 12:57:48
User: N/A
Computer: MYMACHINE
Description:
A process serving application pool 'DefaultAppPool' suffered a fatal communication error with the World Wide Web Publishing Service. The process id was '5764'. The data field contains the error number.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 6d 00 07 80               m..

 

Q: What does exit code '0x800703e9' mean?

A: Searching on the internet or looking it up in Error Lookup we find that 0x800703E9 means "Recursion too deep; the stack overflowed."

Q: Based on the eventlogs, what caused the crash?

A: A stackoverflow exception in the application.  The most likely cause for this is infinite recursion, i.e. function a calls function b calls function a calls function b etc. until the memory allocated for the stack is exhausted.

Reproduce the issue again and gather a memory dump:

1. Restart iis by running iisreset

2. Browse to http://localhost/BuggyBits/CompanyInformation.aspx again without hitting the send button

3. Open a command prompt, move to the debuggers directory and type the following command to attach the debugger and wait for the crash

adplus -crash -pn w3wp.exe -quiet

4. Enter a message on the page and click send

Q: What files get created in the dump folder (i.e. the folder named crash_mode current date and time under the debuggers directory)?

A:

2008-03-04 14:28 4 415 ADPlus_report.txt
2008-03-04 14:28 <DIR> CDBScripts
2008-03-04 14:28 151 874 536 PID-2548__W3WP.EXE__1st_chance_Process_Shut_Down__full_06f0_2008-03-04_14-28-17-145_09f4.dmp
2008-03-04 14:28 111 648 PID-2548__W3WP.EXE__Date_03-04-2008__Time_14-28-0606.log
2008-03-04 14:28 13 156 Process_List.txt

Open the dump to figure out what it is doing:

1. Open the process_shut_down dump (crash_mode with todays date and time in the debuggers directory) in windbg using file/open crash dump

2. Load up the symbols and sos

Examine the stacks:

Note: In a dump file the active thread is the thread that caused the debugger to dump the process, i.e. a process exit exception or other exception. 

One of a few things can cause the process to shut down.

  • An unhandled exception (2nd chance exception), in most cases if this is causing the shutdown, you will also get a 2nd chance exception dump, and this will be the one you want to look at to figure out what caused the crash
  • An external shutdown, i.e. an iisreset, a preemptive recycle (based on recycling options) or someone killing the process from task manager.  If this is the case, the active thread in the process shutdown dump is usually the main thread and you will see a stack similar to the following
    0:000> kL
    ChildEBP RetAddr  
    0014fbfc 7d503faf ntdll!ZwTerminateProcess+0x12
    0014fc38 7d503f5a kernel32!_ExitProcess+0x4b
    0014fc4c 79fd9e8f kernel32!ExitProcess+0x14
    0014fe74 79f7479c mscorwks!SafeExitProcess+0x157
    0014ff10 79004fab mscorwks!HandleExitProcessHelper+0x27
    0014ff10 79004fab mscoree!CorExitProcess+0x46
    0014ff20 77bcaddb mscoree!CorExitProcess+0x46
    0014ff2c 77bcaefb msvcrt!__crtExitProcess+0x29
    0014ff5c 77bcaf52 msvcrt!doexit+0x81
    0014ff70 01001a3c msvcrt!exit+0x11
    0014ffc0 7d4e7d2a w3wp!wmainCRTStartup+0x144
    0014fff0 00000000 kernel32!BaseProcessStart+0x28
    

    If this is the case you should look at the eventlog as it usually contains the recycling reason, and if you are troubleshooting a real crash you should consider disabling recyling while troubleshooting the real crash so that you won't get these type of "red herring" dumps.
  • Something in the process called process exit.  This is usually caused by some type of exception considered fatal, i.e. heap corruption, stack overflow, out of memory exceptions or fatal execution engine exceptions.  In this case the active thread in the process shutdown stack will look (and probably is) completely unrelated to the crash. In this case you will want to look at the log to see what happened just prior to the crash in order to figure out how to proceed and gather new dumps if necessary.

1. Run kb to see which of the above categories this crash/process exit falls into

Q: Which category did it fall into and why?

A: In my particular dump I got this stack for the last remaining executing thread.  Since this is not the main thread (didn't start with w3wp!wmainCRTStartup) this means that the process died and shut down the threads.  The fact that we didn't get a 2nd chance dump means we didnt die from an unhandled exception, and the fact that it wasn't the main thread means that it isn't a normal recycle. 

0:000> kL
ChildEBP RetAddr  
0290fe54 7d4d8c82 ntdll!NtWaitForSingleObject+0x15
0290fec4 79e789c6 kernel32!WaitForSingleObjectEx+0xac
0290ff08 79e7898f mscorwks!CLREventWaitHelper+0x2f
0290ff58 79e78944 mscorwks!CLREvent::WaitEx+0x117
0290ff6c 79efa706 mscorwks!CLREvent::Wait+0x17
0290ff98 79fc8583 mscorwks!SVR::gc_heap::gc_thread_function+0x30
0290ffb8 7d4dfe21 mscorwks!SVR::gc_heap::gc_thread_stub+0x92
0290ffec 00000000 kernel32!BaseThreadStart+0x34
 

Examine the log file to find out what happened just prior to the crash

Q: What events occurred just prior to the crash? 

A: Right before the process exit event I see the following exception (Unknown exception - code e053534f).  Since I don't have the symbol path set up in the environment variables I can't really trust the stack, but what I can see is that we have some .net code (the mscorlib_ni frames) followed by some calls into mscorwks and eventually the kernel32!RaiseException that raises the e053534f exception.  Since this exception code is not a known exception code adplus reports it as unknown exception.  

0:025> g
Tue Mar  4 14:28:16.551 2008 (GMT+1): (9f4.30c): Unknown exception - code e053534f (first chance)
---
--- 1st chance UnknownException exception ----
---------------------------------------------------------------

Occurrence happened at:
Debug session time: Tue Mar  4 15:20:44.687 2008 (GMT+1)
System Uptime: 19 days 12:10:42.146
Process Uptime: 0 days 0:00:17.546
  Kernel time: 0 days 0:00:00.578
  User time: 0 days 0:00:00.640

Faulting stack below ---
 # ChildEBP RetAddr  Args to Child              
WARNING: Stack unwind information not available. Following frames may be wrong.
00 0ff3aecc 7a0ac3d8 e053534f 00000000 00000000 kernel32!RaiseException+0x4e
01 0ff3aee8 79fff01a 0024a900 6ea810e3 0024a900 mscorwks!GetCompileInfo+0x1d896
02 0ff3b06c 79f067d3 06f26a08 6ea81197 06fd9788 mscorwks!NGenCreateNGenWorker+0x213a7
03 0ff3b118 7948d92a 793826f1 06fd98c4 79e82084 mscorwks!GetCLRFunction+0x2371f
04 0ff3b214 793983c8 00000000 06fd975c 00000000 mscorlib_ni+0x3cd92a
05 0ff3b254 793984e3 08000000 00001000 00000001 mscorlib_ni+0x2d83c8
06 0ff3b278 79398550 0ff3b2d8 06fd9470 06f2657c mscorlib_ni+0x2d84e3
07 0ff3b28c 7949959c 00000400 06f26960 00000000 mscorlib_ni+0x2d8550
08 0ff3b2c4 793983c8 06fd8d90 06fd8fe4 06f2657c mscorlib_ni+0x3d959c
09 0ff3b33c 7949959c 00000400 06f26960 00000000 mscorlib_ni+0x2d83c8
0a 00000000 00000000 00000000 00000000 00000000 mscorlib_ni+0x3d959c

We can rebuild the native piece of the stack by using ln (list near) on the mscorwks methods in the process exit dump.
0:000> ln mscorwks!GetCLRFunction+0x2371f
(79f06728)   mscorwks!GetResourceFromDefault+0xb6   |  (79f0681a)   mscorwks!GetResourceStringFromManaged
0:000> ln mscorwks!NGenCreateNGenWorker+0x213a7
(79f0681a)   mscorwks!GetResourceStringFromManaged+0x44   |  (79f069e5)   mscorwks!NativeCompareInfo::CompareStringW
0:000> ln mscorwks!GetCompileInfo+0x1d896
(7a0ac375)   mscorwks!ReportStackOverflow+0x61   |  (7a0ac3dc)   mscorwks!ChangeThreadToSOTolerant
Substituting the unresolved method names (method names with bad symbols) we get this...  
 # ChildEBP RetAddr  Args to Child              
WARNING: Stack unwind information not available. Following frames may be wrong.
00 0ff3aecc 7a0ac3d8 e053534f 00000000 00000000 kernel32!RaiseException+0x4e
01 0ff3aee8 79fff01a 0024a900 6ea810e3 0024a900 mscorwks!ReportStackOverflow+0x61
02 0ff3b06c 79f067d3 06f26a08 6ea81197 06fd9788 mscorwks!GetResourceStringFromManaged+0x44
03 0ff3b118 7948d92a 793826f1 06fd98c4 79e82084 mscorwks!GetResourceFromDefault+0xb6
04 0ff3b214 793983c8 00000000 06fd975c 00000000 mscorlib_ni+0x3cd92a
05 0ff3b254 793984e3 08000000 00001000 00000001 mscorlib_ni+0x2d83c8
06 0ff3b278 79398550 0ff3b2d8 06fd9470 06f2657c mscorlib_ni+0x2d84e3
07 0ff3b28c 7949959c 00000400 06f26960 00000000 mscorlib_ni+0x2d8550
08 0ff3b2c4 793983c8 06fd8d90 06fd8fe4 06f2657c mscorlib_ni+0x3d959c
09 0ff3b33c 7949959c 00000400 06f26960 00000000 mscorlib_ni+0x2d83c8
0a 00000000 00000000 00000000 00000000 00000000 mscorlib_ni+0x3d959c

So we can see that it was a stackoverflow exception that brought the process down.  If we wanted to we could also resolve the top of the .net stack by running !ip2md 790c0000+0x3cd92a for example where 790c0000 is the base address for mscorlib.dll

0:000> !ip2md 790c0000+0x3d959c 
MethodDesc: 7925bc40
Method Name: System.IO.StreamWriter..ctor(System.String)
Class: 7910381c
MethodTable: 7910389c
mdToken: 06003696
Module: 790c2000
IsJitted: yes
m_CodeOrIL: 79499580

But I will leave that as an excercise for you since we only have a short piece of the stack so we don't get deep enough to find the recursion.  Note here that the recursion doesn't happen in the streamwriter constructor.  This is just the last method call when we are running out of stack space as you'll see later 

Configure adplus to gather full dumps on the exception in the log file 

1. Restart iis by running iisreset

2. Browse to http://localhost/BuggyBits/CompanyInformation.aspx again without hitting the send button

3. Copy the attached adplus configuration file to your debuggers directory  (Note: this configuration file causes adplus to take a full dump on the unknown exception)

4. Open a command prompt, move to the debuggers directory and type the following command to attach the debugger and wait for the crash

adplus -pn w3wp.exe -c Unknown.cfg

4. Enter a message on the page and click send

Open the dump to figure out what it is doing:

1. Open the 1st_chance_unknown_exception dump in windbg using file/open crash dump

2. Load up the symbols and sos

Examine the active stack:

1. Run kb 2000 and !clrstack to see what the active thread is doing (i.e. the thread that caused the stackoverflow)

0:014> kb
ChildEBP RetAddr  Args to Child              
0250ae9c 7a0ac3d8 e053534f 00000000 00000000 kernel32!RaiseException+0x53
0250aeb8 79fff01a 001a67c0 c29e1ac8 001a67c0 mscorwks!ReportStackOverflow+0x61
0250b03c 79f067d3 02f367d4 c29e1a1c 0309dfe0 mscorwks!GetResourceStringFromManaged+0x44
0250b0e8 7948d92a 793826f1 0309e11c 79e82084 mscorwks!GetResourceFromDefault+0xb6
0250b1e4 793983c8 00000000 0309dfb4 00000000 mscorlib_ni+0x3cd92a
0250b224 793984e3 08000000 00001000 00000001 mscorlib_ni+0x2d83c8
0250b248 79398550 0250b2a8 0309dcc8 02f35fd0 mscorlib_ni+0x2d84e3
0250b25c 7949959c 00000400 02f3640c 00000000 mscorlib_ni+0x2d8550
0250b294 793983c8 0309d5e8 0309d83c 02f35fd0 mscorlib_ni+0x3d959c
0250b30c 7949959c 00000400 02f3640c 00000000 mscorlib_ni+0x2d83c8
00000000 00000000 00000000 00000000 00000000 mscorlib_ni+0x3d959c

0:014> !clrstack
OS Thread Id: 0x177c (14)
ESP       EIP     
0250b090 7d4e2366 [HelperMethodFrame_2OBJ: 0250b090] System.Environment.GetResourceFromDefault(System.String)
0250b0f0 7948d92a System.IO.__Error.WinIOError(Int32, System.String)
0250b11c 79395557 System.IO.FileStream.Init(System.String, System.IO.FileMode, System.IO.FileAccess, Int32, Boolean, System.IO.FileShare, Int32, System.IO.FileOptions, SECURITY_ATTRIBUTES, System.String, Boolean)
0250b214 793983c8 System.IO.FileStream..ctor(System.String, System.IO.FileMode, System.IO.FileAccess, System.IO.FileShare, Int32, System.IO.FileOptions)
0250b240 793984e3 System.IO.StreamWriter.CreateFile(System.String, Boolean)
0250b250 79398550 System.IO.StreamWriter..ctor(System.String, Boolean, System.Text.Encoding, Int32)
0250b270 7949959c System.IO.StreamWriter..ctor(System.String)
0250b27c 0fe90b52 Utility.WriteToLog(System.String, System.String)
0250b2b0 0fe90b0b ExceptionHandler.LogException(System.Exception)
...
0252f0c4 0fe90bd4 Utility.WriteToLog(System.String, System.String)
0252f170 0fe90b0b ExceptionHandler.LogException(System.Exception)
0252f174 0fe90bd4 Utility.WriteToLog(System.String, System.String)
0252f220 0fe90b0b ExceptionHandler.LogException(System.Exception)
0252f224 0fe90a81 BuggyMail.SendEmail(System.String, System.String)
0252f2c8 0fe90a28 CompanyInformation.btnContact_Click(System.Object, System.EventArgs)
0252f2d8 6619004e System.Web.UI.WebControls.Button.OnClick(System.EventArgs)
0252f2ec 6619023c System.Web.UI.WebControls.Button.RaisePostBackEvent(System.String)
0252f300 661901b8 System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(System.String)
0252f304 6614b47c System.Web.UI.Page.RaisePostBackEvent(System.Web.UI.IPostBackEventHandler, System.String)
0252f30c 6614b3d2 System.Web.UI.Page.RaisePostBackEvent(System.Collections.Specialized.NameValueCollection)
0252f31c 6614e263 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
0252f518 6614d8c3 System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
0252f550 6614d80f System.Web.UI.Page.ProcessRequest()
0252f588 6614d72f System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)
0252f590 6614d6c2 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
0252f5a4 0fe90205 ASP.companyinformation_aspx.ProcessRequest(System.Web.HttpContext)
0252f5a8 65fe6bfb System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
0252f5dc 65fe3f51 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
0252f61c 65fe7733 System.Web.HttpApplication+ApplicationStepManager.ResumeSteps(System.Exception)
0252f66c 65fccbfe System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object)
0252f688 65fd19c5 System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest)
0252f6bc 65fd16b2 System.Web.HttpRuntime.ProcessRequestNoDemand(System.Web.HttpWorkerRequest)
0252f6c8 65fcfa6d System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr, Int32)
0252f8d8 79f047fd [ContextTransitionFrame: 0252f8d8] 
0252f90c 79f047fd [GCFrame: 0252f90c] 
0252fa68 79f047fd [ComMethodFrame: 0252fa68] 

 

Q: Can you tell what the issue is from here?

A:  From !clrstack it appears that we are processing companyinformation.aspx and in the page_load method it calls into BuggyMail.SendEmail.  From there we go into some type of recursion between ExceptionHandler.LogException(System.Exception) and Utility.WriteToLog(System.String, System.String), and eventually we run out of stackspace and throw the stackoverflow exception that kills the process...  To resolve the issue we need to figure out a way to stop the recursive loop between LogException and WriteToLog.  You may be tempted to believe that it is the System.IO.__Error.WinIOError or the GetResourceFromDefault methods that cause the stackoverflow but in reality those methods are just the straws that broke the camels back... They are called in each

Verify your assumption in the code and modify the code to resolve the issue 

1. Open ExceptionHandler.cs and Utility.cs and device a plan for how to fix the problem

    public static void LogException(Exception ex)
    {
        Utility.WriteToLog(ex.Message, "c:\\log.txt");
    }

    public static void WriteToLog(string message, string fileName)
    {
        try
        {
            using (StreamWriter sw = new StreamWriter(fileName))
            {
                //Log the event with date and time.
                sw.WriteLine("--------------------------");
                sw.WriteLine(DateTime.Now.ToLongTimeString());
                sw.WriteLine("-------------------");
                sw.WriteLine(message);
            }
        }
        catch (Exception ex)
        {
            ExceptionHandler.LogException(ex);
        }
    }

A: From this and the stack we can backtrack what happened... We got an exception in BuggyMail.SendMail and asked to log the Exception which calls WriteToLog, unfortunately we also got an exception in WriteToLog when actually writing to the log (for those of you interested, you can find these exceptions with !dso and the reason we get an exception in WriteToLog is because we (the w3wp.exe process) don't have access to write to c:\log.txt.  The solution shold be to have some other exceptionhandling mechanism in WriteToLog.

Have fun,

Tess

Attachment: unknown.cfg
  • PingBack from http://blogs.msdn.com/tess/pages/net-debugging-demos-information-and-setup-instructions.aspx

  • Sharepoint No VSE WSS for VS 2008 Until Summer [Via: Tariq ] WPF Podder v2 has been released! [Via:...

  • I'm facing an issue with a StackOverflow problem, I get the dump, but when I load it it says "unable to verify the checksum for System.Data.ni.dll" or something similar, no matter what I do, I've tried updating to .NET 2 SP1, regenerating the native images, I've made sure it reads the correct files (filemon), I've ran windbg directly from the server, etc

    The weird thing is that this only happens on the server though, we actually have reproduced the issue in two servers, but we cannot get the same error on the developer machines

    these are server 2003 with service pack 2 that have .NET 2, 3 and 3.5 installed

    when I run the windbg commands everything points to an issue in System.Data.dll, but since the symbols for that file are not loaded I don't get a lot of information, I get a bunch of calls to system.data.ni.dll -

    how can I get the right symbols file?

    any other clues you might have?

    thanks for all these tutorials Tess, I really appreciate all your work

  • There are no symbols for the native images (.ni) and you dont really need them.  You will get the calls on the stack if you use !clrstack.

    Tess

  • this is the output running kb and !clrstack

    0:026> kb 2000

    ChildEBP RetAddr  Args to Child              

    072d3664 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x114f8b

    072d40c0 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072d4b1c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072d5578 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072d5fd4 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072d6a30 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072d748c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072d7ee8 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072d8944 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072d93a0 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072d9dfc 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072da858 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072db2b4 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072dbd10 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072dc76c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072dd1c8 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072ddc24 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072de680 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072df0dc 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072dfb38 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e0594 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e0ff0 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e1a4c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e24a8 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e2f04 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e3960 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e43bc 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e4e18 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e5874 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e62d0 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e6d2c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e7788 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e81e4 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e8c40 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072e969c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072ea0f8 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072eab54 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072eb5b0 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072ec00c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072eca68 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072ed4c4 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072edf20 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072ee97c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072ef3d8 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072efe34 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f0890 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f12ec 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f1d48 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f27a4 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f3200 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f3c5c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f46b8 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f5114 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f5b70 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f65cc 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f7028 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f7a84 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f84e0 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f8f3c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072f9998 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072fa3f4 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072fae50 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072fb8ac 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072fc308 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072fcd64 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072fd7c0 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072fe21c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072fec78 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    072ff6d4 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    07300130 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    07300b8c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    073015e8 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    07302044 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    07302aa0 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    073034fc 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    07303f58 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    073049b4 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    07305410 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    07305e6c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    073068c8 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    07307324 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    07307d80 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    073087dc 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    07309238 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    07309c94 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    0730a6f0 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    0730b14c 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    0730bba8 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    0730c604 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    0730d060 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    0730dabc 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    0730e518 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    0730ef84 652857b0 00000000 00000200 18612264 System_Data_ni!_load_config_used+0x1140e0

    0730efd0 65218285 00000000 186ed6b4 000000bc System_Data_ni!_load_config_used+0x1140e0

    0730f030 651feb12 00000092 14307e20 1c6b0de4 System_Data_ni!_load_config_used+0xa6bb5

    0730f164 79e880df 01bf1f0e 0730f198 84adef90 System_Data_ni!_load_config_used+0x8d442

    0730f124 081684e7 0730f198 062b43e0 00000001 mscorwks!PreStubWorker+0x139

    *** WARNING: Unable to verify checksum for System.Web.RegularExpressions.ni.dll

    WARNING: Frame IP not in any known module. Following frames may be wrong.

    0730f1b0 66f12980 1c0579b0 18312720 6628efd2 0x81684e7

    00000000 00000000 00000000 00000000 00000000 System_Web_RegularExpressions_ni+0x12980

    0:026> !clrstack

    OS Thread Id: 0x1740 (26)

    ESP       EIP    

    072d2c3c 6528665b System.Data.BinaryNode.EvalBinaryOp(Int32, System.Data.ExpressionNode, System.Data.ExpressionNode, System.Data.DataRow, System.Data.DataRowVersion, Int32[])

    that's all BCL, where do I go from there?

  • the windows event I get is this:

    EventType clr20r3, P1 w3wp.exe, P2 6.0.3790.3959, P3 45d6968e, P4 system.data, P5 2.0.0.0, P6 471ebf27, P7 1840, P8 0, P9 system.stackoverflowexception, P10 NIL.

    and this:

    A process serving application pool 'ProcessName' suffered a fatal communication error with the World Wide Web Publishing Service. The process id was '5664'. The data field contains the error number.

    8007006d

  • Eber, the stackoverflow you are getting with the System.Data.BinaryNode.EvalBinaryOp is a known issue that occurrs when you have an extremely long filter i.e. condtionx and conditiony and conditionz etc.

    basically for each and or or it goes into a separate eval of the left site and eval of the right site recursively, so that is why you get a stackoverflow.  Run !dso to see the condition and look to see if it is meant to be like that or if you have some logic issues that cause the filter exception to be set like that inadvertedly.  I don't have access to my tools at the moment but when I get back into the office I can write a post describing why it happens and more details about it.

  • When you use very complex Rowfilters or expressions on datasets or datatables you may end up getting

  • Eber, I just wrote a post about the issue on http://blogs.msdn.com/tess/archive/2008/03/31/net-case-study-stackoverflow-exception-when-using-a-complex-rowfilter.aspx if you want to have a look

  • thanks Tess, the weird thing is that it only happens on the server, not on any of the development machines

    admittedly we use some complex expressions

    I'll check the new article, thanks a lot

  • I cannot run 0:025> g, Why?

    Why should you create the full dump?

    Thanks

  • I did not follow the stpes "Examine the log file to find out what happened just prior to the crash

    Q: What events occurred just prior to the crash?  

    "

    Could you explain it with more details?

  • H Debugger,

    0:025> g

    is something you would find in the log file created by adplus...

    as a sidenote: the 0:025> meant that it was on thread 25,  the g means go...   in a dump you can't go anymore so you can't run g at the prompt unfortunately...

    ____________________________

    regarding "I did not follow the stpes "Examine the log file to find out what happened just prior to the crash

    Q: What events occurred just prior to the crash?  

    "  

    During the time you are attached in crash mode, the debugger will keep spitting out debugging info to the log file based on your adplus configuration.  This means that at the time of the crash you can look in the log file and have a look at what debugging info it had spit out just prior to the crash, in order to see what events led up to the crash...

  • I think there could be an infinite loop like:

    private void function_name(Leave_Application_Table lat)

       {

          function_name();

       }

  • ------------------------------------------------------------------------------------

    unfortunately we also got an exception in WriteToLog when actually writing to the log (for those of you interested, you can find these exceptions with !dso and the reason we get an exception in WriteToLog is because we (the w3wp.exe process) don't have access to write to c:\log.txt.

    ----------------------------------------------------------------------------------------

    Unfortunately,I can not reproduce the bug.Because the ASPNET account has access to write to the c:\logtxt.:((

Page 1 of 2 (26 items) 12
Leave a Comment
  • Please add 1 and 5 and type the answer here:
  • Post