Speaking of which... : Legacy

Walkthrough - Troubleshooting a native memory leak

JohanS — Thu, 21 Feb 2008 18:38:50 GMT

Problem:

A customer called in. They had a Web Service running on a single IIS6. Memory usage would slowly increase and not be released. As a workaround they'd currently set the application pool to recycle at 500 MB, causing a few failed requests upon each restart.

I thought I'd describe how I went about troubleshooting this scenario.

The first dump

The customer had already taken a memory dump using DebugDiag, so obviously I asked them to upload it. It was a huge dump. 1.64 GB! I opened it up in windbg and ran the following command:

!eeheap -gc

I haven't mentioned !eeheap before. It gives me a nice summary of what's on the managed heap(s).

0:000> !eeheap -gc
Number of GC Heaps: 4
------------------------------
Heap 0 (0x000dfaf0)
generation 0 starts at 0x1124b1c0
generation 1 starts at 0x111c3684
generation 2 starts at 0x102d0030
ephemeral segment allocation context: none
segment    begin       allocated     size
0x102d0000 0x102d0030 0x1165033c 0x138030c(20,448,012)
Large object heap starts at 0x202d0030
segment    begin       allocated     size
0x202d0000 0x202d0030 0x20636810 0x003667e0(3,565,536)
Heap Size 0x16e6aec(24,013,548)
------------------------------
Heap 1 (0x000dfee8)
generation 0 starts at 0x15365b5c
generation 1 starts at 0x152f2138
generation 2 starts at 0x142d0030
ephemeral segment allocation context: none
segment    begin       allocated     size
0x142d0000 0x142d0030 0x1577d230 0x14ad200(21,680,640)
Large object heap starts at 0x252d0030
segment    begin       allocated     size
0x252d0000 0x252d0030 0x25391190 0x000c1160(790,880)
Heap Size 0x156e360(22,471,520)
------------------------------
Heap 2 (0x000e09f8)
generation 0 starts at 0x19480e94
generation 1 starts at 0x193ec5b0
generation 2 starts at 0x182d0030
ephemeral segment allocation context: none
segment    begin       allocated     size
0x182d0000 0x182d0030 0x1985ed34 0x158ed04(22,605,060)
Large object heap starts at 0x222d0030
segment    begin       allocated     size
0x222d0000 0x222d0030 0x222e8748 0x00018718(100,120)
Heap Size 0x15a741c(22,705,180)
------------------------------
Heap 3 (0x000e14e0)
generation 0 starts at 0x1d0f887c
generation 1 starts at 0x1d056738
generation 2 starts at 0x1c2d0030
ephemeral segment allocation context: none
segment    begin       allocated     size
0x1c2d0000 0x1c2d0030 0x1d56af68 0x129af38(19,509,048)
Large object heap starts at 0x232d0030
segment    begin       allocated     size
0x232d0000 0x232d0030 0x23319578 0x00049548(300,360)
Heap Size 0x12e4480(19,809,408)
------------------------------
GC Heap Size 0x54e06e8(88,999,656)

So, as we can see the managed memory is pretty evenly distributed in the 4 managed heaps. If you look down at the bottom you'll see that total memory usage is at 88 MB. (This is a figure I would have gotten by running !dumpheap -stat as well. So in case you're wondering; that would have worked just as fine.) Anyway, if only ~90 MB is in the managed heap, then the remaining 1.55 GB must be on the native heap.

New dumps

Troubleshooting the native heap is a bit harder. The sos extension and windbg is a really nice team so working without them can be tough. Fortunately there is a nice feature included in DebugDiag that lets it perform an automatic analysis. This is usually the best way to get information from a native leak. Sometimes you may want to get a little additional information from windbg as well, but most of the time DebugDiag's auto analysis will get you a long way.

LeakTrack

DebugDiag comes with a handy little dll called LeakTrack. LeakTrack attaches to the process in question and monitors memory allocations and their related call stacks. Since the dll is injected into the process it can have a slight impact on performance, but if you want to troubleshoot the process you really have no other options.

There are two ways to start monitoring a process with LeakTrack:

Create a "Memory and Handle Leak"-rule using the wizard normally displayed upon start up.
Cancel the wizard. Go to the "Processes"-tab. Right-click the w3wp.exe you wish to monitor and select "Monitor for leaks".

LeakTrack will begin monitoring from the moment you inject it. Obviously, it won't be able to get any data for events that have already occurred, so injecting LeakTrack when memory usage is already at it's peak won't really do much good. Instead I usually recycle the process and attach LeakTrack as soon as possible.

This time I chose to let the customer create a "Memory and Handle Leak"-rule using the wizard. This rule then created dumps at certain intervals which the customer uploaded to me.

Analyzing the dumps

So the next step is to let DebugDiag analyze the dumps. Here's how you do it:

Run DebugDiag
Cancel the wizard
Go to the "Advanced Analysis"-tab
"Add Data Files" - point to the dump
Choose "Memory Pressure Analysis" in the "Available Analysis Scripts" list.
Make sure your dump is selected in the "Data Files" list.
"Start Analysis"

The usual suspects

Once DebugDiag has finished analyzing the dump you will get a new browser window with a report. The report I got showed the following right at the top:

So what does this mean? It means that LeakTrack has monitored the allocations made by a certain third party dll. It has also monitored if that memory has been properly released. All in all the .dll now has almost 150 MB of allocations that are unaccounted for. It would be safe to assume that this is our culprit. Further down in the report we have a link called "Leak analysis report". Clicking it will bring us to the following graphical representation:

As you can see the .dll in question is right at the top. But wait a minute! According to the graph mscorsvr has ~9 MB of outstanding allocations as well. Does this mean that mscorsvr is leaking too, though not as much?

The answer is no. As I mentioned before; LeakTrack will monitor all allocations and check that the memory is properly released as well. All we know is that mscorsvr has allocated 8,75 MB of memory that hasn't been released yet. There's nothing to suggest it won't release it eventually. The odds of the stray 148 MB eventually being released, on the other hand, are a lot slimmer in my opinion.

What causes the leak?

Since the dll responsible for the allocations is third party I can't find out exactly what causes the leak within the third party dll. What I can find out is what calls in the customer's code lead us down the leaking code path in the dll. LeakTrack monitors the callstacks related to memory allocations. Looking at the report I was able to find some nice callstacks that the customer should keep an eye on. It might be possible for them to tweak their code slightly in order to avoid the scenario.

3rdparty!IFMX_IsInTransaction+5671
3rdparty!SQLEndTran+e456
odbc32!SQLFetch+160
[DEFAULT] [hasThis] Boolean System.Data.Odbc.OdbcDataReader.Read()
[DEFAULT] [hasThis] I4 System.Data.Common.DbDataAdapter.FillLoadDataRow(Class System.Data.Common.SchemaMapping)
[DEFAULT] [hasThis] I4 System.Data.Common.DbDataAdapter.FillFromReader(Object,String,Class System.Data.IDataReader,I4,I4,Class System.Data.DataColumn,Object)
[DEFAULT] [hasThis] Void System.Data.DataSet.set_EnforceConstraints(Boolean)
[DEFAULT] [hasThis] I4 System.Data.Common.DbDataAdapter.Fill(Class System.Data.DataSet,String,Class System.Data.IDataReader,I4,I4)
[DEFAULT] [hasThis] Void System.Data.DataSet.set_EnforceConstraints(Boolean)
[DEFAULT] [hasThis] I4 System.Data.Common.DbDataAdapter.FillFromCommand(Object,I4,I4,String,Class System.Data.IDbCommand,ValueClass System.Data.CommandBehavior)
[DEFAULT] [hasThis] I4 System.Data.Common.DbDataAdapter.Fill(Class System.Data.DataSet,I4,I4,String,Class System.Data.IDbCommand,ValueClass System.Data.CommandBehavior)
[DEFAULT] [hasThis] I4 System.Data.Common.DbDataAdapter.Fill(Class System.Data.DataSet)
[DEFAULT] [hasThis] Class System.Data.DataSet Company.App.Server.DAL.DAO.Informix.AbstractDAO.Fill(String,String,I4)
[DEFAULT] [hasThis] String Company.App.Server.DAL.DAO.Informix.ResourceDAO.SelectResourcePriceType(String,String)
[DEFAULT] [hasThis] Class System.Data.DataSet Company.App.Server.DAL.DAO.Informix.AbstractDAO.Fill(String,String,I4)
[DEFAULT] [hasThis] Void Company.App.Server.DAL.DAO.Informix.ResourceDAO..ctor(ByRef Class System.Data.IDbConnection,ByRef Class System.Data.IDbTransaction)
[DEFAULT] [hasThis] String Company.App.Server.Services.Pricing.PriceFinder.GetPriceType(String,String,ByRef Class System.Data.IDbConnection,Class System.Data.IDbTransaction)
[DEFAULT] [hasThis] Class Company.App.Core.Query.Result.PriceResult Company.App.Server.Services.Pricing.PriceFinder.GetResourceDiscountPrices(String,String,ValueClass System.DateTime,String,SZArray R4,String,String,ByRef Class System.Data.IDbConnection,Class System.Data.IDbTransaction)
[DEFAULT] [hasThis] String Company.App.Server.Services.Pricing.PriceFinder.GetPriceType(String,String,ByRef Class System.Data.IDbConnection,Class System.Data.IDbTransaction)
[DEFAULT] [hasThis] Class Company.App.Core.Query.Result.PriceResult Company.App.Server.Facade.BookingDataSource.GetResourceDiscountPrices(String,String,ValueClass System.DateTime,String,SZArray R4,String,String)
[DEFAULT] [hasThis] Class Company.App.Core.Query.Result.PriceResult App.BookingData.GetResourceDiscountPrices(String,String,ValueClass System.DateTime,String,SZArray R4,String,String)
[DEFAULT] [hasThis] Class Company.App.Core.Query.Result.PriceResult Company.App.Server.Facade.BookingDataSource.GetResourceDiscountPrices(String,String,ValueClass System.DateTime,String,SZArray R4,String,String)
mscorsvr!CallDescrWorker+30
mscorsvr!COMMember::InvokeMethod+95a
mscorsvr!COMMember::CreateInstance+358
[DEFAULT] [hasThis] SZArray Object System.Web.Services.Protocols.LogicalMethodInfo.Invoke(Object,SZArray Object)
[DEFAULT] [hasThis] Void System.Web.Services.Protocols.WebServiceHandler.Invoke()
[DEFAULT] [hasThis] SZArray Object System.Web.Services.Protocols.LogicalMethodInfo.Invoke(Object,SZArray Object)
[DEFAULT] [hasThis] Void System.Web.Services.Protocols.WebServiceHandler.CoreProcessRequest()
[DEFAULT] [hasThis] Void System.Web.Services.Protocols.WebServiceHandler.Invoke()
[DEFAULT] [hasThis] Void System.Web.Services.Protocols.SyncSessionlessHandler.ProcessRequest(Class System.Web.HttpContext)
[DEFAULT] [hasThis] Void System.Web.Services.Protocols.WebServiceHandler.CoreProcessRequest()
[DEFAULT] [hasThis] Void System.Web.HttpApplication/CallHandlerExecutionStep.System.Web.HttpApplication+IExecutionStep.Execute()
[DEFAULT] [hasThis] Class System.Exception System.Web.HttpApplication.ExecuteStep(Class IExecutionStep,ByRef Boolean)
[DEFAULT] [hasThis] Void System.Web.HttpApplication.ResumeSteps(Class System.Exception)
[DEFAULT] [hasThis] Class System.Exception System.Web.HttpApplication.ExecuteStep(Class IExecutionStep,ByRef Boolean)
[DEFAULT] [hasThis] Class System.IAsyncResult System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(Class System.Web.HttpContext,Class System.AsyncCallback,Object)
[DEFAULT] [hasThis] Void System.Web.HttpApplication.ResumeSteps(Class System.Exception)
[DEFAULT] [hasThis] Void System.Web.HttpRuntime.ProcessRequestInternal(Class System.Web.HttpWorkerRequest)
[DEFAULT] Void System.Web.HttpRuntime.ProcessRequest(Class System.Web.HttpWorkerRequest)
[DEFAULT] [hasThis] Void System.Web.HttpRuntime.ProcessRequestInternal(Class System.Web.HttpWorkerRequest)
[DEFAULT] [hasThis] I4 System.Web.Hosting.ISAPIRuntime.ProcessRequest(I,I4)
[DEFAULT] Void System.Web.HttpRuntime.ProcessRequest(Class System.Web.HttpWorkerRequest)
mscorsvr!ComToComPlusWorker+1e0
mscorsvr!ComToComPlusWorker_Wrapper+38
mscorsvr!Thread::DoADCallBack+5c
mscorsvr!ComToComPlusWorker+65
mscorsvr!ComCallWrapper::NeedToSwitchDomains+24
aspnet_isapi!HttpCompletion::ProcessRequestInManagedCode+17e
aspnet_isapi!HttpCompletion::ProcessCompletion+24
aspnet_isapi!CorThreadPoolWorkitemCallback+13
mscorsvr!ThreadpoolMgr::ExecuteWorkRequest+19
mscorsvr!ThreadpoolMgr::WorkerThreadStart+129
mscorsvr!ThreadpoolMgr::intermediateThreadProc+44
kernel32!GetModuleHandleForUnicodeString+20
kernel32!GetModuleHandleForUnicodeString+20
kernel32!BasepGetModuleHandleExW+17f
ntdll!LdrpGetProcedureAddress+b3
ntdll!LdrpGetProcedureAddress+b3
ntdll!LdrpCallInitRoutine+14
ntdll!LdrpInitializeThread+18f
ntdll!LdrpInitializeThread+18f
ntdll!ZwContinue+c
kernel32!BaseThreadStart+34

Okay, now what?

Well, this is actually more or less where the road ends for me. Had the problem lied within a Microsoft component it would have been up to us to fix it. Had it been within the customer's code I would have been able to use the customers symbols to get a more detailed callstack and provide him with pointers on how to resolve it as well, but since this is a third party component the next step is to forward it to the third-party vendor.

/ Johan

My language of choice can't decode an URIEncoded string

JohanS — Fri, 16 Mar 2007 11:25:00 GMT

Problem:

Some time ago I was asked to help in decoding an URI Encoded string from VB6. URL Decoding is simple enough, but when you're working with Greek, Hebrew, etc. that just isn't enough.

Resolution:

A quick search revealed a truckload of samples on how to do URL Decoding, but URI Decoding is a bit harder. I decided to stand on the shoulders of geniuses and ran Reflector on the Microsoft.JScript.dll. The code was fairly easy to convert, but I thought I'd share it in case you don't want to do it yourself. :)

I've commented out (but otherwise left intact) all the spots where .NET would throw an exception, since this would be a good spot to implement your own error handler.

Anyway, bring on the code!

/ Johan

Private Function Encode2(ByVal Text1 As String) As String
      Dim builder1 As String
      builder1 = ""
      Dim num1 As Integer
      num1 = 0
      Do While (num1 < Len(Text1))
            Dim ch1 As String
            ch1 = Mid(Text1, num1 + 1, 1)
            If InURISet(ch1, 2) Then
                  builder1 = builder1 & (ch1)
            Else
                  Dim num2 As Integer
                  num2 = AscW(ch1)
                  If ((num2 >= 0) And (num2 <= 127)) Then
                        Call AppendInHex(builder1, num2)
                  ElseIf ((num2 >= 128) And (num2 <= 2047)) Then
                        Call AppendInHex(builder1, (RShiftWord(num2, 6) Or 192))
                        Call AppendInHex(builder1, ((num2 And 63) Or 128))
                  ElseIf ((num2 < 55296) Or (num2 > 57343)) Then
                        Call AppendInHex(builder1, (RShiftWord(num2, 12) Or 224))
                        Call AppendInHex(builder1, ((RShiftWord(num2, 6) And 63) Or 128))
                        Call AppendInHex(builder1, ((num2 And 63) Or 128))
                  Else
                        If ((num2 >= 56320) And (num2 <= 57343)) Then
'                              Throw New JScriptException(JSError.URIEncodeError)
                        End If
                        If (num1 >= Len(Text1)) Then
'                              Throw New JScriptException(JSError.URIEncodeError)
                        End If
                        Dim num3 As Integer
                        num3 = Mid(Text1, num1 + 1, 1)
                        If ((num3 < 56320) Or (num3 > 57343)) Then
'                              Throw New JScriptException(JSError.URIEncodeError)
                        End If
                        num2 = ((LShiftWord((num2 - 55296), 10) + num3) + 9216)
                        Call AppendInHex(builder1, (RShiftWord(num2, 18) Or 240))
                        Call AppendInHex(builder1, ((RShiftWord(num2, 12) And 63) Or 128))
                        Call AppendInHex(builder1, ((RShiftWord(num2, 6) And 63) Or 128))
                        Call AppendInHex(builder1, ((num2 And 63) Or 128))
                  End If
            End If
            num1 = num1 + 1
      Loop
      Encode2 = builder1
End Function

Private Function Decode2(ByVal Text1 As String) As String
      Dim builder1 As String
      Dim num1 As Integer
      num1 = 0
      Do While (num1 < Len(Text1))
            Dim ch1 As String
            ch1 = Mid(Text1, (num1) + 1, 1)
            If (ch1 <> "%") Then
                  builder1 = builder1 & (ch1)
            Else
                  Dim ch2 As String
                  Dim num2 As Integer
                num2 = num1
                  If ((num1 + 2) >= Len(Text1)) Then
'                        Throw New JScriptException(JSError.URIDecodeError)
                  End If
                  Dim num3 As Byte
                  num3 = HexValue(Mid(Text1, (num1 + 2), 1), Mid(Text1, (num1 + 3), 1))
                  num1 = (num1 + 2)
                  If ((num3 And 128) = 0) Then
                        ch2 = Chr(num3)
                  Else
                        Dim num4 As Integer
                        num4 = 1
                        Do While ((LShiftWord(num3, (num4 And 31)) And 128) <> 0)
                              num4 = num4 + 1
                        Loop
                        If (((num4 = 1) Or (num4 > 4)) Or ((num1 + ((num4 - 1) * 3)) >= Len(Text1))) Then
'                              Throw New JScriptException(JSError.URIDecodeError)
                        End If
                        Dim num5 As Integer
                        num5 = (num3 And RShiftWord(255, ((num4 + 1) And 31)))
                        Do While (num4 > 1)
                              If (Mid(Text1, (num1 + 2), 1) <> "%") Then
'                                    Throw New JScriptException(JSError.URIDecodeError)
                              End If
                              num3 = HexValue(Mid(Text1, (num1 + 3), 1), Mid(Text1, (num1 + 4), 1))
                              num1 = (num1 + 3)
                              If ((num3 And 192) <> 128) Then
'                                    Throw New JScriptException(JSError.URIDecodeError)
                              End If
                              num5 = ((LShiftWord(num5, 6)) Or (num3 And 63))
                              num4 = num4 - 1
                        Loop
                        If ((num5 >= 55296) And (num5 < 57344)) Then
'                              Throw New JScriptException(JSError.URIDecodeError)
                        End If
                        If (num5 < 65536) Then
                              ch2 = (ChrW(num5))
                        Else
                              If (num5 > 1114111) Then
'                                    Throw New JScriptException(JSError.URIDecodeError)
                              End If
                              builder1 = builder1 & (Chr(((RShiftWord((num5 - 65536), 10) And 1023) + 55296)))
                              builder1 = builder1 & (Chr((((num5 - 65536) And 1023) + 56320)))
                              GoTo Label_01D4
                        End If
                  End If
                  If InURISet(ch2, 0) Then ' This can probably be omitted. It looks like it'll never be True.
                        builder1 = builder1 & Mid(Text1, num2 + 1, ((num1 - num2) + 1))
                  Else
                        builder1 = builder1 & ch2
                  End If
Label_01D4:
            End If
            num1 = num1 + 1
      Loop
      Decode2 = builder1
End Function

Private Function HexValue(ByVal ch1 As String, ByVal ch2 As String) As Byte
      Dim num1 As Integer
      Dim num2 As Integer
      num1 = HexDigit(ch1)
      num2 = HexDigit(ch2)
      If ((num1 < 0) Or (num2 < 0)) Then
'            Throw New JScriptException(JSError.URIDecodeError)
      End If
      HexValue = CByte((LShiftWord(num1, 4) Or num2))
End Function

Private Function HexDigit(ByVal c As String) As Integer
    Dim retVal As Integer
    retVal = -1
    If ((c >= "0") And (c <= "9")) Then
          retVal = (Asc(c) - Asc("0"))
    End If
    If ((c >= "A") And (c <= "F")) Then
          retVal = (((10) + Asc(c)) - Asc("A"))
    End If
    If ((c >= "a") And (c <= "f")) Then
          retVal = (((10) + Asc(c)) - Asc("a"))
    End If
    HexDigit = retVal
End Function

Function LShiftWord(ByVal w As Integer, ByVal c As Integer) As Integer
LShiftWord = w * (2 ^ c)
End Function

Function RShiftWord(w As Integer, c As Integer) As Integer
RShiftWord = w \ (2 ^ c)
End Function

Private Function InURISet(ByVal ch As String, ByVal flags As Integer) As Boolean
    Dim bRetval As Boolean
    bRetval = False
    If (flags = 2) Then
        If ((((ch >= "0") And (ch <= "9")) Or ((ch >= "A") And (ch <= "Z"))) Or ((ch >= "a") And (ch <= "z"))) Then
            bRetval = True
        End If
        Select Case ch
              Case "_", "~", "'", "(", ")", "*", "-", ".", "!"
                    bRetval = True
        End Select
    End If
    If (flags = 1) Then
        Select Case ch
              Case "#", "$", "&", "+", ",", "/", ":", ";", "=", "?", "@"
                    bRetval = True
        End Select
    End If
    InURISet = bRetval
End Function

Private Sub AppendInHex(ByRef bs As String, ByVal value As Integer)
      bs = bs & "%"
      Dim num1 As Integer
      num1 = (RShiftWord(value, 4) And 15)
      bs = bs & (IIf((num1 >= 10), Chr((num1 - 10) + 65), Chr(num1 + 48)))
      num1 = (value And 15)
      bs = bs & (IIf((num1 >= 10), Chr((num1 - 10) + 65), Chr(num1 + 48)))
End Sub