One of my customers had a problem on his IIS server, where the IIS process hosting his web application was suffering from very high memory utilization over time. Eventually the process got so high in memory that the web application became completely unresponsive and incoming requests were no longer getting served. Based on these details, we began troubleshooting the issue as a memory leak/high memory issue.
The first step was to capture a memory dump of the problematic process, when its memory usage was at its worst point. The customer had actually already captured the memory dump before calling, so we moved to the next step immediately: data analysis.
NOTE: The customer had not used the Debug Diagnostics tool to capture the dump file. The Debug Diagnostics tool is able to capture specific memory leak data which would have made the analysis much easier. However, in this instance we were still able to get to the root cause even without the leak tracking functionality.
The dump files showed the CRuntime heap growing with various allocation sizes and various number of allocations. If we looked specifically at the allocation groups, we could see that some contained data while others contained pointers to objects. Generally, large allocations contained the data (such as a 100k ASP response) and smaller allocations contained pointers (such as a pointer to a script object). Here is a snippet from one of the larger allocations and the data it contained:
030c2f00 18036524 00000000 0315cfd8 00000000 $e..............030c2f10 00030007 00080101 09c242e0 140bc5b8 .........B......030c2f20 00000000 00000000 0b3efb7c 00000042 ........|.>.B...030c2f30 00000005 1c284fc8 a15ac8d4 00000000 .....O(...Z.....030c2f40 030d7178 52495748 00070039 00080101 xq..HWIR9.......030c2f50 6b658ba8 6b658b90 6b658b7c 6b658b64 ..ek..ek|.ekd.ek030c2f60 6b658b4c 6b658b38 6b658b28 6b658af4 L.ek8.ek(.ek..ek030c2f70 6b658ae4 6b658ad4 6b658ac4 6b658aac ..ek..ek..ek..ekdc 0x030c33b0+0x8030c33b8 18036524 00000000 09c31ab8 06a8d500 $e..............030c33c8 00000000 00000000 00040003 00080101 ................030c33d8 18036524 00000000 0a09ba90 00000000 $e..............030c33e8 00030007 00080101 41d8ff20 00000000 ........ ..A....030c33f8 00000000 00000000 0b62065c 0000004a ........\.b.J...030c3408 00000000 6bba0068 1a858c47 00000000 ....h..kG.......030c3418 00000800 00000000 00070005 00080101 ................030c3428 00000190 00000000 0311a750 00000000 ........P.......dc 0x030c33d0+0x8030c33d8 18036524 00000000 0a09ba90 00000000 $e..............030c33e8 00030007 00080101 41d8ff20 00000000 ........ ..A....030c33f8 00000000 00000000 0b62065c 0000004a ........\.b.J...030c3408 00000000 6bba0068 1a858c47 00000000 ....h..kG.......030c3418 00000800 00000000 00070005 00080101 ................030c3428 00000190 00000000 0311a750 00000000 ........P.......030c3438 00000000 030c4f20 00000000 00000000 .... O..........030c3448 00050003 00080101 18036524 00000000 ........$e......
In the large allocations we could actually see the data in a readable format. It was an ASP script sample, from one of the ASP template objects. But looking at the smaller allocations we saw that the data was in HEX, and there was a repeating 1st parameter of 18036524. This first parameter gave us a good clue as to what allocated the memory; looking at the location that the memory address pointed to, we see it was a function of the component Scriptproxy.dll:
From this data we could see that a component called scriptproxy.dll appeared to be allocating memory to store instances of its class object, and appeared to be populating its class objects with the ASP template data. Unfortunately, the memory it was allocating was not ever being released, which was resulting in the high memory.
Because Scriptproxy.dll is not a Microsoft component, we were not able to discern from the memory dump why it was acting this way, nor whether this was expected behavior. The easiest was to resolve the customer’s high memory problem quickly was to remove the Scriptproxy.dll component and disable it from loading in the IIS processes. Once the customer did that, the memory problem no longer occurred. I then suggested to the customer that they follow up with the vendor of Scriptproxy.dll for further assistance, since really they would be the experts and would know why Scriptproxy.dll was behaving this way. The lm command inside the windbg debugger can be used to show details about a component, so I ran it against Scriptproxy to get the vendor info:
0:000> lmvm scriptproxystart end module name18030000 1803b000 scriptproxy (deferred) Image path: C:\Program Files\Network Associates\VirusScan\scriptproxy.dllImage name: scriptproxy.dllTimestamp: Thu Nov 11 12:31:26 2004 (4193CC1E)CheckSum: 00000000ImageSize: 0000B000File version: 8.0.0.955Product version: 8.0.0.0File flags: 0 (Mask 3F)File OS: 40004 NT Win32File type: 2.0 DllFile date: 00000000.00000000Translations: 0000.04b0CompanyName: Network Associates, Inc.ProductName: VirusScan EnterpriseProductVersion: 8.0.0FileVersion: 8.0.0.955PrivateBuild: VSE.8.0.0.955 F1FileDescription: VirusScan Script ProxyLegalCopyright: Copyright© 1995-2004 Networks Associates Technology, Inc. All Rights Reserved.
My customer did get back to me after speaking with the vendor, and it turns out that they had seen these types of issues before with Scriptproxy when it was run on server application platforms that relied on script processing. They also pointed to some publicly available documentation, https://kc.mcafee.com/corporate/index?page=content&id=KB55963, to assist our customer further.