AsiaTech: Microsoft APGC Internet Developer Support Team

We focus on various troubleshooting plan and solution on IIS web platform and distributed applications

Induced GC happened frequently and caused High CPU when creating Serviced Component in ASP.NET 2.0

Induced GC happened frequently and caused High CPU when creating Serviced Component in ASP.NET 2.0

  • Comments 1

 

Customer reported one High CPU issue while one ASP.NET application is under normal load.

 

Regarding  ASP.NET application, besides the multiple threads busy, tighten looping code defect  some general possibilities, looking at the memory usage is quite important because heavy memory pressure may lead to frequent Garbage collection, and this is a kind of expensive operation for CPU resource.  If GC often occurs  (we verify this through %Time in GC and Induced GC performance counters under .NET CLR Memory Object), then we need to dig out how it happens in this pattern.

 

 Check the Performance monitor, the Induced GC increased quickly and turns to be the culprit of the high CPU issue:

 

 

Usually, Induced GC is caused by explicit GC.Collect calls in customer's code, however, after double confirmed with customer's application team and reviewing the code, the code doesn't do such a explicit call at all.

 

We noticed this is one application which uses Enterprise Serviced Components, thus recommend customer using Dispose instead of DisposeObject:

 

http://msdn.microsoft.com/en-us/library/system.enterpriseservices.servicedcomponent.disposeobject.aspx

 

After implementing this, the performance didn't get better.

 

By analyzing memory dump thoroughly, we finally found out how the Induced GC happened in this way. Here is the call stack which triggered GC:

 

Unmanaged Call Stack:

 

Child-SP          RetAddr           Call Site

00000000`0f59bbe8 00000000`7704ce00 ntdll!ZwWaitForSingleObject+0xa

00000000`0f59bbf0 000007fe`f52fa74a kernel32!WaitForSingleObjectEx+0x9c

00000000`0f59bcb0 000007fe`f52fa83b mscorwks!CLREventWaitHelper+0x42

00000000`0f59bd10 000007fe`f53cee38 mscorwks!CLREvent::WaitEx+0x63

00000000`0f59bdc0 000007fe`f53c1cb7 mscorwks!SVR::gc_heap::wait_for_gc_done+0x88

00000000`0f59be00 000007fe`f542913b mscorwks!SVR::GCHeap::GarbageCollectGeneration+0x147

00000000`0f59be90 000007fe`f5412ac3 mscorwks!SVR::GCHeap::GarbageCollect+0x5b

00000000`0f59bee0 000007fe`f57301a5 mscorwks!GCInterface::AddMemoryPressure+0x13b

00000000`0f59bf70 000007fe`f57b9beb mscorwks!RCW::AddMemoryPressure+0x15

00000000`0f59bfb0 000007fe`f57dbbf4 mscorwks!RCW::CreateRCW+0x17b

00000000`0f59c030 000007fe`f57dbdea mscorwks!COMInterfaceMarshaler::CreateObjectRef+0x74

00000000`0f59c130 000007fe`f58843f4 mscorwks!COMInterfaceMarshaler::WrapWithComObject+0x3a

00000000`0f59c1a0 000007fe`f0268995 mscorwks!MarshalNative::WrapIUnknownWithComObject+0x134

00000000`0f59c3c0 000007fe`f0270933 System_EnterpriseServices_ni!System.EnterpriseServices.RemoteServicedComponentProxy..ctor(System.Type, IntPtr, Boolean)+0xd5

00000000`0f59c420 000007fe`f0270213 System_EnterpriseServices_ni!System.EnterpriseServices.FastRSCPObjRef.GetRealObject(System.Runtime.Serialization.StreamingContext)+0x33

00000000`0f59c470 000007fe`f37254db System_EnterpriseServices_ni!System.EnterpriseServices.ServicedComponentProxyAttribute.CreateProxy(System.Runtime.Remoting.ObjRef, System.Type, System.Object, System.Runtime.Remoting.Contexts.Context)+0x133

00000000`0f59c500 000007fe`f37253ff mscorlib_ni!System.Runtime.Remoting.RemotingServices.SetOrCreateProxy(System.Runtime.Remoting.Identity, System.Type, System.Object)+0x9b

00000000`0f59c560 000007fe`f372466f mscorlib_ni!System.Runtime.Remoting.RemotingServices.GetOrCreateProxy(System.Runtime.Remoting.Identity, System.Object, Boolean)+0xbf

00000000`0f59c5c0 000007fe`f027050a mscorlib_ni!System.Runtime.Remoting.RemotingServices.InternalUnmarshal(System.Runtime.Remoting.ObjRef, System.Object, Boolean)+0x12f

00000000`0f59c650 000007fe`f40e3edf System_EnterpriseServices_ni!System.EnterpriseServices.ServicedComponentProxyAttribute.CreateInstance(System.Type)+0x2ba

00000000`0f59c7c0 000007fe`f5416e61 mscorlib_ni!System.Runtime.Remoting.Activation.ActivationServices.IsCurrentContextOK(System.Type, System.Object[], Boolean)+0x9ba81f

 

Managed call Stack:

 

Child-SP         RetAddr          Call Site

000000000f59c3c0 000007fef0270933 System.EnterpriseServices.RemoteServicedComponentProxy..ctor(System.Type, IntPtr, Boolean)

000000000f59c420 000007fef0270213 System.EnterpriseServices.FastRSCPObjRef.GetRealObject(System.Runtime.Serialization.StreamingContext)

000000000f59c470 000007fef37254db System.EnterpriseServices.ServicedComponentProxyAttribute.CreateProxy(System.Runtime.Remoting.ObjRef, System.Type, System.Object, System.Runtime.Remoting.Contexts.Context)

000000000f59c500 000007fef37253ff System.Runtime.Remoting.RemotingServices.SetOrCreateProxy(System.Runtime.Remoting.Identity, System.Type, System.Object)

000000000f59c560 000007fef372466f System.Runtime.Remoting.RemotingServices.GetOrCreateProxy(System.Runtime.Remoting.Identity, System.Object, Boolean)

000000000f59c5c0 000007fef027050a System.Runtime.Remoting.RemotingServices.InternalUnmarshal(System.Runtime.Remoting.ObjRef, System.Object, Boolean)

000000000f59c650 000007fef40e3edf System.EnterpriseServices.ServicedComponentProxyAttribute.CreateInstance(System.Type)

 

 

When Serviced Component is being created, .NET RCW layer will call GC.AddMemoryPressure  in the mscorwks!RCW::CreateRCW  call. When the memory resource is under pressure, it will add the Induced GC counter, and then call GarbageCollect. this means the increased Induced GC is from this part.

 

To resolve this issue, we need to reduce the frequency of creating Serviced Component instances in the application. Suggested customer to create a pool of the Serviced Component, to reduce the frequency of component creation.  And based on our testing, before creating Serviced Component, add RemoveMemoryPressure(4004) will also help this issue as a workaround. 4004 is a defined size for local serviced component call used by AddMemoryPressure  in the above logic.

 

Customer finally created a pool of the Serviced Component, and the application runs very well under the heavy load.

 

Regards,

Freist Li from APGC DSI Team

Leave a Comment
  • Please add 3 and 7 and type the answer here:
  • Post
  • Please let me know how you have generated the memory dump.

Page 1 of 1 (1 items)