April, 2009

  • Never doubt thy debugger

    StackOverflowException and DataBind()


    The application pool for this site was getting disabled quite frequently and we found quite a few entries like the following in the event log:

    Event Type: Error
    Event Source: W3SVC
    Event Category: None
    Event ID: 1002
    Date: 19/11/2008
    Time: 15:20:23
    User: N/A
    Computer: <computername>
    Description: Application pool 'DefaultAppPool' is being automatically disabled due to a series of failures in the process(es) serving that application pool

    In this case the customer already had some debugging skills so when he called CSS he already had a few dumps available to analyze; they were some process shutdown dumps (taken on Kernel32!TerminateProcess), strangely we had no second chance dumps (maybe because of one of these reasons?) but they’ve been a good point to start from anyway.

    First, let’s have a look at the exceptions:

    0:000> !dumpheap -type Exception -stat 


    MT Count TotalSize Class Name
    79333ed4 1 12 System.Text.DecoderExceptionFallback
    79333e90 1 12 System.Text.EncoderExceptionFallback
    79330d44 1 72 System.ExecutionEngineException
    79330cb4 1 72 System.StackOverflowException
    79330c24 1 72 System.OutOfMemoryException
    7931ffd4 1 72 System.NullReferenceException
    6610c7fc 1 84 System.Web.HttpUnhandledException
    79330dd4 2 144 System.Threading.ThreadAbortException
    79318afc 2 144 System.InvalidOperationException
    7931740c 5 160 System.UnhandledExceptionEventHandler
    Total 16 objects

    Remember, a few special exceptions are loaded when the AppDomain is first created (see here) so let’s try to see if there is still something significant on the threads:

    0:000> !threads
    ThreadCount: 21
    UnstartedThread: 0
    BackgroundThread: 21
    PendingThread: 0
    DeadThread: 0
    Hosted Runtime: no
    PreEmptive GC Alloc Lock
    ID OSID ThreadOBJ State GC Context Domain Count APT Exception
    17 1 950 000f2580 1808220 Enabled 00000000:00000000 000eed48 0 MTA (Threadpool Worker)
    27 2 1bfc 00100018 b220 Enabled 00000000:00000000 000eed48 0 MTA (Finalizer)
    28 3 300 00117068 80a220 Enabled 00000000:00000000 000eed48 0 MTA (Threadpool Completion Port)
    29 4 1cf4 0011a458 1220 Enabled 00000000:00000000 000eed48 0 Ukn
    9 5 ec0 001702e0 220 Enabled 00000000:00000000 000eed48 0 Ukn
    31 6 2414 00171f68 180b220 Enabled 00000000:00000000 000eed48 0 MTA (Threadpool Worker)
    32 7 1f24 00172338 180b220 Enabled 00000000:00000000 000eed48 0 MTA (Threadpool Worker)
    33 8 97c 001729e0 180b220 Enabled 00000000:00000000 000eed48 0 MTA (Threadpool Worker)
    34 9 13bc 00173088 180b220 Enabled 00000000:00000000 000eed48 0 MTA (Threadpool Worker)
    35 a 22f8 00173a50 180b220 Enabled 00000000:00000000 000eed48 0 MTA (Threadpool Worker)
    36 b 1e60 00174540 180b220 Disabled 12445a64:124475b8 0011ab48 1 MTA (Threadpool Worker) System.StackOverflowException (1a0a00a4)
    37 c 1c8c 00174cf8 180b220 Enabled 00000000:00000000 000eed48 0 MTA (Threadpool Worker)
    15 d 1934 0017d040 880a220 Enabled 00000000:00000000 000eed48 0 MTA (Threadpool Completion Port)
    7 e 2028 0017ece8 220 Enabled 00000000:00000000 000eed48 0 Ukn
    8 f dbc 0016f280 220 Enabled 00000000:00000000 000eed48 0 Ukn
    4 10 2298 05007008 220 Enabled 00000000:00000000 000eed48 0 Ukn
    6 11 20c0 05004278 220 Enabled 00000000:00000000 000eed48 0 Ukn
    3 12 c9c 05003b70 220 Enabled 00000000:00000000 000eed48 0 Ukn
    5 13 1a84 04ff7a60 220 Enabled 00000000:00000000 000eed48 0 Ukn
    2 14 253c 0d378ee0 220 Enabled 00000000:00000000 000eed48 0 Ukn
    42 15 1760 0d38b3d8 220 Enabled 00000000:00000000 000eed48 0 Ukn

    Dumping the managed stack for thread 36, the recursion is quite obvious:

    0:036> !clrstack
    OS Thread Id: 0x1e60 (36)
    04e94acc 77e42004 [FaultingExceptionFrame: 04e94acc]
    04e95058 77e42004 [HelperMethodFrame: 04e95058]
    04e950c4 65230cc1 System.Data.RBTree`1+TreePage[[System.__Canon, mscorlib]]..ctor(Int32)
    04e950d4 65230abe System.Data.RBTree`1[[System.__Canon, mscorlib]].AllocPage(Int32)
    04e950f8 652309fc System.Data.RBTree`1[[System.__Canon, mscorlib]].InitTree()
    04e95108 65230961 System.Data.DataRowCollection..ctor(System.Data.DataTable)
    04e95118 652307c2 System.Data.DataTable..ctor()
    04e95134 65230535 System.Data.Common.DataTableMapping.GetDataTableBySchemaAction(System.Data.DataSet, System.Data.MissingSchemaAction)
    04e95150 6523037c System.Data.ProviderBase.SchemaMapping..ctor(System.Data.Common.DataAdapter, System.Data.DataSet, System.Data.DataTable, System.Data.ProviderBase.DataReaderContainer, Boolean, System.Data.SchemaType, System.String, Boolean, System.Data.DataColumn, System.Object)
    04e9519c 6522fbac System.Data.Common.DataAdapter.FillMappingInternal(System.Data.DataSet, System.Data.DataTable, System.String, System.Data.ProviderBase.DataReaderContainer, Int32, System.Data.DataColumn, System.Object)
    04e951d0 6522fc1e System.Data.Common.DataAdapter.FillMapping(System.Data.DataSet, System.Data.DataTable, System.String, System.Data.ProviderBase.DataReaderContainer, Int32, System.Data.DataColumn, System.Object)
    04e95218 6522f9e6 System.Data.Common.DataAdapter.FillFromReader(System.Data.DataSet, System.Data.DataTable, System.String, System.Data.ProviderBase.DataReaderContainer, Int32, Int32, System.Data.DataColumn, System.Object)
    04e95270 6522f942 System.Data.Common.DataAdapter.Fill(System.Data.DataSet, System.String, System.Data.IDataReader, Int32, Int32)
    04e952b4 65230105 System.Data.Common.DbDataAdapter.FillInternal(System.Data.DataSet, System.Data.DataTable[], Int32, Int32, System.String, System.Data.IDbCommand, System.Data.CommandBehavior)
    04e9530c 65230010 System.Data.Common.DbDataAdapter.Fill(System.Data.DataSet, Int32, Int32, System.String, System.Data.IDbCommand, System.Data.CommandBehavior)
    04e95350 6559401d System.Data.Common.DbDataAdapter.Fill(System.Data.DataSet, System.String)
    04e95384 6678d11f System.Web.UI.WebControls.SqlDataSourceView.ExecuteSelect(System.Web.UI.DataSourceSelectArguments)
    04e95424 6668f498 System.Web.UI.DataSourceView.Select(System.Web.UI.DataSourceSelectArguments, System.Web.UI.DataSourceViewSelectCallback)
    04e95434 66792c0f System.Web.UI.WebControls.DataBoundControl.PerformSelect()
    04e9544c 6679285e System.Web.UI.WebControls.BaseDataBoundControl.DataBind()
    04e95458 667dcf35 System.Web.UI.WebControls.GridView.DataBind()
    04e9545c 660e1ac3 System.Web.UI.WebControls.BaseDataBoundControl.EnsureDataBound()
    04e95488 668ecdcd System.Web.UI.WebControls.BaseDataBoundControl.set_RequiresDataBinding(Boolean)
    04e95498 667da23b System.Web.UI.WebControls.GridView.set_PageIndex(Int32)
    04e954a8 0ad3e096 ASP.search_aspx.Gridview1_DataBound(System.Object, System.EventArgs)
    04e959a8 667928ce System.Web.UI.WebControls.BaseDataBoundControl.OnDataBound(System.EventArgs)
    04e959bc 66792b45 System.Web.UI.WebControls.DataBoundControl.OnDataSourceViewSelectCallback(System.Collections.IEnumerable)
    04e959cc 6668f4a4 System.Web.UI.DataSourceView.Select(System.Web.UI.DataSourceSelectArguments, System.Web.UI.DataSourceViewSelectCallback)
    04e959dc 66792c0f System.Web.UI.WebControls.DataBoundControl.PerformSelect()
    04e959f4 6679285e System.Web.UI.WebControls.BaseDataBoundControl.DataBind()
    04e95a00 667dcf35 System.Web.UI.WebControls.GridView.DataBind()
    04e95a04 660e1ac3 System.Web.UI.WebControls.BaseDataBoundControl.EnsureDataBound()
    04e95a30 668ecdcd System.Web.UI.WebControls.BaseDataBoundControl.set_RequiresDataBinding(Boolean)
    04e95a40 667da23b System.Web.UI.WebControls.GridView.set_PageIndex(Int32)

    04e95a50 0ad3e096 ASP.search_aspx.Gridview1_DataBound(System.Object, System.EventArgs)
    04e95f50 667928ce System.Web.UI.WebControls.BaseDataBoundControl.OnDataBound(System.EventArgs)
    04e95f64 66792b45 System.Web.UI.WebControls.DataBoundControl.OnDataSourceViewSelectCallback(System.Collections.IEnumerable)
    04e95f74 6668f4a4 System.Web.UI.DataSourceView.Select(System.Web.UI.DataSourceSelectArguments, System.Web.UI.DataSourceViewSelectCallback)
    04e95f84 66792c0f System.Web.UI.WebControls.DataBoundControl.PerformSelect()
    04e95f9c 6679285e System.Web.UI.WebControls.BaseDataBoundControl.DataBind()
    04e95fa8 667dcf35 System.Web.UI.WebControls.GridView.DataBind()
    04e95fac 660e1ac3 System.Web.UI.WebControls.BaseDataBoundControl.EnsureDataBound()
    04e95fd8 668ecdcd System.Web.UI.WebControls.BaseDataBoundControl.set_RequiresDataBinding(Boolean)
    04e95fe8 667da23b System.Web.UI.WebControls.GridView.set_PageIndex(Int32)
    04e95ff8 0ad3e096 ASP.search_aspx.Gridview1_DataBound(System.Object, System.EventArgs)

    04ebea70 0ad3e096 ASP.search_aspx.Gridview1_DataBound(System.Object, System.EventArgs)
    04ebef70 667928ce System.Web.UI.WebControls.BaseDataBoundControl.OnDataBound(System.EventArgs)
    04ebef84 66792b45 System.Web.UI.WebControls.DataBoundControl.OnDataSourceViewSelectCallback(System.Collections.IEnumerable)
    04ebef94 6668f4a4 System.Web.UI.DataSourceView.Select(System.Web.UI.DataSourceSelectArguments, System.Web.UI.DataSourceViewSelectCallback)
    04ebefa4 66792c0f System.Web.UI.WebControls.DataBoundControl.PerformSelect()
    04ebefbc 6679285e System.Web.UI.WebControls.BaseDataBoundControl.DataBind()
    04ebefc8 667dcf35 System.Web.UI.WebControls.GridView.DataBind()
    04ebefcc 660e1ac3 System.Web.UI.WebControls.BaseDataBoundControl.EnsureDataBound()
    04ebeff8 660e1a57 System.Web.UI.WebControls.BaseDataBoundControl.OnPreRender(System.EventArgs)
    04ebf008 667e07da System.Web.UI.WebControls.GridView.OnPreRender(System.EventArgs)
    04ebf030 660abc21 System.Web.UI.Control.PreRenderRecursiveInternal()
    04ebf048 660abc7c System.Web.UI.Control.PreRenderRecursiveInternal()
    04ebf060 660abc7c System.Web.UI.Control.PreRenderRecursiveInternal()
    04ebf078 660abc7c System.Web.UI.Control.PreRenderRecursiveInternal()
    04ebf090 660abc7c System.Web.UI.Control.PreRenderRecursiveInternal()
    04ebf0a8 660a7c4b System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
    04ebf200 660a77a4 System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
    04ebf238 660a76d1 System.Web.UI.Page.ProcessRequest()
    04ebf270 660a7666 System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)
    04ebf27c 660a7642 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
    04ebf290 0ad37afe ASP.search_aspx.ProcessRequest(System.Web.HttpContext)
    04ebf2a0 660adb16 System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
    04ebf2d4 6608132c System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
    04ebf314 6608c5c3 System.Web.HttpApplication+ApplicationStepManager.ResumeSteps(System.Exception)
    04ebf364 660808ac System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object)
    04ebf380 66083e1c System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest)
    04ebf3b4 66083ac3 System.Web.HttpRuntime.ProcessRequestNoDemand(System.Web.HttpWorkerRequest)
    04ebf3c4 66082c5c System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr, Int32)
    04ebf5d8 79f68cde [ContextTransitionFrame: 04ebf5d8]
    04ebf60c 79f68cde [GCFrame: 04ebf60c]
    04ebf768 79f68cde [ComMethodFrame: 04ebf768]


    The next step is to find out why the OnDataBound event is firing over and over; here is the source code for the faulting page (simplified to show only the relevant bits):

    Sub Page_Load(ByVal Sender As Object, ByVal E As EventArgs)

    Dim cmd As String = "select * from products"

    If Not IsPostBack Then

    ' when there is a blank search, display all products
    If SearchString = "" Then

    SqlDataSource1.SelectCommand = cmd

    ' when there are keywords (not a blank search)

    ' some logic to change the sql query

    If GridView1.Rows.Count < 3 Then ' displays suggestions if results are less then 3
    For i = 0 To Keywords.Count - 1
    ' some logic to change the sql query

    For i = 0 To FoundNewKeywords.Count - 1
    For s = 0 To Keywords.Count - 1
    ' some logic to change the sql query

    For j = i To FoundNewKeywords.Count - 1
    ' some logic to change the sql query

    If j = FoundNewKeywords.Count - 1 And FoundNewKeywords.Count > 2 Then
    For m = j - 1 To 1 Step -1
    ' some logic to change the sql query

    If i < 1 Then
    For m = 2 To j - 1
    ' some logic to change the sql query
    End If
    End If
    End If
    End If
    End If
    End Sub

    Sub Gridview1_DataBound(ByVal sender As Object, ByVal e As EventArgs)
    ' Some custom logic here
    End Sub


    From a high level perspective the intention was to start with a “return everything” query and refine it step by step taking into account keywords specified by the user and add logic to suggest other records that might interest the user. From an operation standpoint calling DataBind() multiple times means the whole data binding logic is executed over and over, such as the custom DataBound event handler: that way, depending on what we’re doing, we could exhaust the stack space which is what was happening in this case.

    The solution is simple: there is no need to call DataBind() many times, just call it once after we have refined our query.




    Quote of the day:
    To avoid situations in which you might make mistakes may be the biggest mistake of all. - Peter McWilliams
  • Never doubt thy debugger

    Where’s my dump gone?!?


    Have you ever found yourself patiently waiting for a problem to reproduce with your debugger ready, and when it happens you just find there are no signs of your dump anywhere? If you are using adplus you likely had the text logs, but nothing more…

    In such situations it is possible that the OS is terminating the process before the dump is fully written. If you are tying to dump an IIS process and you want to save some time and headaches, try this small cunning: temporarily disable the “Enable pinging” and “Enable rapid-fail protection” flags in the Health tab for your application pool:

     Health tab

    Sometimes can be useful to tell IIS to not kill a failing worker process but rather leave it orphaned:

    Features of Worker Process Isolation Mode

    Orphaning Worker Processes

    You can configure worker process isolation mode to orphan a worker process that the WWW service deems to be failing. The WWW service usually terminates a failing worker process and replaces it. If you enable orphaning, the WWW service allows a failing worker process to continue running, but separates it from the application pool (making it an orphan) and starts a new worker process in its place. You can configure the WWW service to run a command on the orphaned worker process — for example, launching a debugger.

    For more information about orphaning worker processes, including how to configure this feature, see Running IIS 6.0 as an Application Server in this book and OrphanWorkerProcess Metabase Property metabase property.

    In IIS 6 there the following three metabase properties can be handy to run some custom actions in case of an application pool failure:

    • OrphanWorkerProcess: The OrphanWorkerProcess property, when set to true, notifies the World Wide Web Publishing Service (WWW Service) not to terminate a worker process that fails to respond to pings, but to instead orphan the worker process in the application pool if the worker process suffers fatal errors
    • OrphanActionParams: The OrphanActionParams property specifies command-line parameters for the executable specified by the OrphanActionExe Metabase Property. To specify the process ID of the orphaned process, use %1%.
    • OrphanActionExe: The OrphanActionExe property specifies an executable to run when the World Wide Web Publishing Service (WWW service) orphans a worker process. You can use the OrphanActionParams Metabase Property to send parameters to the executable

    The article How to generate a dump file when ASP.NET deadlocks in IIS 6.0 explains how to use the properties above and despite its title, you can use this technique to run the custom actions you need/want not only in case of an ASP.NET deadlock but in every circumstance that fits your needs.

    In IIS 7 you can easily use the Advanced Settings dialog or your application pool:

     Advanced Settings

    Finally, while w3wp.exe cannot run without Data Execution Prevention, if you are debugging a custom process you may want to add it to the exclusion list:

    Data Execution Prevention




    Quote of the day:
    If the only tool you have is a hammer, you tend to see every problem as a nail. - Abraham Maslow
  • Never doubt thy debugger

    WPF, 3D and services: supported (again)?


    I wrote about why GDI+ is not supported in a service a couple of years ago but this is still a debated topic (or I should better say a misunderstood topic), then WPF (Windows Presentation Foundation) came into the game and it brought some more uncertainties with it.

    Recently I worked on a custom application which basically was meant to made of a Windows Form client and a WCF Web Service used to get some data from a database, create a 3D image (a sort of chart) and send it to the client as a jpg image; the graphic part was done using classes such as Viewport3D, PerspectiveCamera, ModelVisual3D, RenderTargetBitmap and others taken from the System.Windows.Media.*  and System.Drawing namespaces.

    Everything was working fine as long as the project was being developed and debugged against the ASP.NET Development Server (Cassini), but when the project was deployed to IIS7 the image returned was missing the 3D part and the fancy transparency and shadow effects added through WPF classes, it merely had a blue background, nothing more.  

    Advanced Settings

    The first thing I thought to, are the differences between Cassini and IIS: the former is a process (not a service), it runs under the credentials of the logged on user and also has Administrator rights as Visual Studio does (or should, to work properly). In IIS7 is really easy to test if this was a permission or missing user desktop issue, I changed the Identity property for the application pool to run it under my domain account and changed the Load User Profile to true to have access the my system variables, registry etc…

    No luck.

    I’ll save you the other tests I’ve done, what is important to know is that there actually is another difference between IIS and Cassini on Windows 2008 / Vista, or I should better say one of the differences I have mentioned before has an important implication on those platforms: services now run in an isolated session (Session 0) while processes run in the session assigned to the user logged on the machine:

    Application Compatibility: Session 0 Isolation

    In Windows XP®, Windows Server® 2003, and earlier versions of the Windows® operating system, all services run in the same session as the first user who logs on to the console. This session is called Session 0. Running services and user applications together in Session 0 poses a security risk because services run at elevated privilege and therefore are targets for malicious agents that are looking for a means to elevate their own privilege levels.

    The Windows Vista® and Windows Server® 2008 operating systems mitigate this security risk by isolating services in Session 0 and making Session 0 non-interactive. In Windows Vista and Windows Server 2008, only system processes and services run in Session 0. The first user logs on to Session 1, and subsequent users log on to subsequent sessions. This approach means that services never run in the same session as users' applications and are therefore protected from attacks that originate in application code.

    Specific examples of affected driver classes include:

    • Printer drivers, which are loaded by the spooler service
    • All drivers authored with the User Mode Driver Framework (UMDF) because these drivers are hosted by a process in Session 0

    Application classes affected by this feature include:

    • Services that create UI
    • A service that tries to use window-message functions such as SendMessage and PostMessage to communicate with an application
    • Applications creating globally named objects

    This can easily been verified adding the “Session ID” column in Task Manager: as you can see, the WinForm client (Test3DInService.exe) is running in Session 2, w3wp.exe is running in Session 0 (despite the fact that the process is running as my domain account and it has loaded the user profile) and finally WebDev.WebServer.exe (Cassini) is running in Session 2:

     Windows Isolation

    Even more important is this document: Impact of Session 0 Isolation on Services and Drivers in Windows Vista:

    Because Session 0 is no longer a user session in Windows Vista, services that are running in Session 0 do not have access to the video driver in Windows Vista. This means that any attempt that a service makes to render graphics fails. In current builds of Windows Vista, querying the display resolution and color depth in Session 0 reports the correct results for the system up to a maximum of 1920x1200 at 32 bits per pixel (bpp).

    Got it!

    Well, almost, but we’re pretty close. I said the blue background was actually created, and other simple 2D shapes could be successfully rendered as well, so the statement “any attempt that a service makes to render graphic fails” is not fully correct, is should read as “any attempt that a service makes to render graphics that relies on the video driver fails”. Not all graphics need to interact with the video driver so generally speaking, GDI can be used (that’s a different set of libraries than GDI+) and “simple” graphic rendering should normally work fine (as if did in my sample with 2D graphics): the trick is never call (or try to call) into the video driver. This unfortunately means the fancy WPF shadows, transparencies and effects are off limits from a service. It is possible to use BIDs and BIDSection.

    I could think to 3 possible solutions (in order of my preference) to this case, not all of them very nice but if nothing else can be done… if you have anything to add, feel free Smile:

    • If it is possible to move the logic to create the 3D image, the Web Service call could just return to the client the data needed to build the image and do the “hard work” in WPF either in a Windows Forms client or in Silverlight if you wish
    • Create a sort of custom web server using the hostable web core provided by IIS 7, embed it into a usermode process (not in a service) and use it to run your code and serve the 3D image: this means there must be an account always logged on the server to keep this process running
    • Rely only on “safe and simple” drawing objects or work directly with bytes structures to design your graphic “point by point”: this is the hard way of doing graphics and you’ll need to code it in C++ I think, you’ll likely have to renounce using effects such as transparencies but at least it should work


    Quote of the day:
    A compromise is the art of dividing a cake in such a way that everyone believes he has the biggest piece. - Ludwig Erhard
Page 1 of 1 (3 items)