ASP.NET Case Study: Bad perf, high memory usage and high CPU in GC - Death By ViewState

re: ASP.NET Case Study: Bad perf, high memory usage and high CPU in GC - Death By ViewState

AndrewSeven — Fri, 24 Nov 2006 20:41:10 GMT

Most cases where I have seen excessive viewstate, there have been one or many things on the page that have viewstate enabled, but which don't actual use the viewstate on postback.

Its not always so easy, but selectively turning off viewstate on some controls goes a long way towards keeping it under control.

I find it unfortunate that I can't easily turn off viewstate "by default" an then selectively turn it on for certain pages or turn it off "by default" for a page and only turn it on for some controls. I always have to go the other way, leaving it on globally and trying to turn it off case by case.

(I haven't dug into this in 2.0 to see if its the same as 1.x )

re: ASP.NET Case Study: Bad perf, high memory usage and high CPU in GC - Death By ViewState

Chris Love — Fri, 24 Nov 2006 20:59:34 GMT

This is so true. ViewState is a real performance killer. I am doing some refinement of how I handle ViewState, http://professionalaspnet.com/blogs/aspnet_blog/archive/2006/11/16/Persist-the-ViewState-and-Increase-ASP.NET-Performance.aspx. I plan on posting an update to this in the next week or so. One thing I have learned is to not use the GridView Control and minimize the use of any other Web Control in a grid format or on the page itself. I had a client call me in at the last day before a deadline because they were dumping about 30 MB into viewstate so there would be no postback to process data. Similar to your scenario. I told them to just re-architech and take an extra week.

re: ASP.NET Case Study: Bad perf, high memory usage and high CPU in GC - Death By ViewState

Ted Jardine — Fri, 24 Nov 2006 21:03:53 GMT

Picky? I hardly call that picky ;-)

Interesting Finds: Week of Thanksgiving 1

Jason Haley — Mon, 27 Nov 2006 00:23:57 GMT

Interesting Finds: Week of Thanksgiving 1

Jason Haley — Mon, 27 Nov 2006 00:30:06 GMT

Death by ViewState

Weblog di Fabio Cozzolino — Mon, 27 Nov 2006 22:08:52 GMT

Uno dei blog che maggiormente seguo è quello di Tess Ferrandezprincipalmente incentrato su pratiche di

Link Listing - November 28, 2006

Christopher Steen — Wed, 29 Nov 2006 07:24:10 GMT

Common Gotcha: Don't forget to when adding providers [Via: ScottGu ] Mounting ISO Images with Vista...

re: ASP.NET Case Study: Bad perf, high memory usage and high CPU in GC - Death By ViewState

Milan Negovan — Fri, 01 Dec 2006 18:26:47 GMT

Thank you for another great post, Tess!

ASP.NET 2.0 has these really cool "page state persisters" which allow you to store view state elsewhere (see http://www.aspnetresources.com/blog/page_state_persisters_overview.aspx).

Even though we're always on the lookout for ways to cut view state in our pages, at times you can't escape it, so we chose to store it in the database.

re: ASP.NET Case Study: Bad perf, high memory usage and high CPU in GC - Death By ViewState

Seth — Tue, 06 Feb 2007 19:44:49 GMT

Hi,

I have a question about high memory usage. We have an issue with one of our SQL servers that results in data not being returned to the application in a timely mannor. when this happens, the connection pool count increases, the number of asp.net requests current increases, and in some cases, the process size jumps from ~200 MB to 400 - 600 MB. In several cases, I have taken a process dump and observed the following from a dumpheap -stat:

MT Count TotalSize Class Name

0x000fd210 444 458,348,564 Free

it seems to be high memory usage allocated by 'Free' - is there a way I can determine what's going on here.

thanks.

Seth

re: ASP.NET Case Study: Bad perf, high memory usage and high CPU in GC - Death By ViewState

Tess — Wed, 07 Feb 2007 10:37:30 GMT

Wow, that is extremely large free sections.

There is two common ways that you end up with those

If you dump out !dumpheap -mt 0x00fd210 -min 100000 to get the ones over 100 k, and then compare the addresses of the objects with the output of !eeheap -gc you can check if they fall in one of the small object segments or on the large object segments

1. If they are on the LOH you have probably allocated a lot of large objects at the same time and just deallocated them. Since not much if any relocation is done on the LOH you can get a lot of fragmentation here.

You can run dc on some of the addresses to get a feel for what used to be here, or look at other large objects to get a pattern.

2. If they are on the small object heap, perhaps you are pinning a lot. Look at the amount of pinned objects in perfmon for the process or run !gchandles to see how many pinned objects there are. Since you can't relocated when pinning you'll get a lot of fragmentation then, but in 2.0 that is not so common.

/Tess

High CPU in GC and other badness caused by SetProcessAffinityMask

If broken it is, fix it you should — Mon, 12 Mar 2007 18:31:19 GMT

I thought I'd share a support story with you from a very interesting case I have. My customer is running

re: ASP.NET Case Study: Bad perf, high memory usage and high CPU in GC - Death By ViewState

.rip — Mon, 03 Sep 2007 06:55:27 GMT

Hi,

I confronted with that problem too. But, I don't understand, why Microsoft's developers didn't fixed this problem. I wrote scipt for "adplus" and run his when CPU-loading was about 100%. Body of script:

c:\windows\symbols

</Sympath>

CRASH

</RunMode>

Quiet

</Option>

</Settings>

</PreCommands>

NoDumpOnFirstChance

</Option>

</Exceptions>

<NewBP>

mscorsvr!gc_heap::allocate_large_object

</Address>

<Type>

</Type>

Stack

</Actions>

!clrstack;

</CustomActions>

</ReturnAction>

</NewBP>

</Breakpoints>

</ADPlus>

Then, I analyzed LOG-file and selected method calls which repeated many times:

Current stack below ---

# ChildEBP RetAddr Args to Child

00 02b5f170 792117f6 00014c6c 00000000 00014c6c mscorsvr!gc_heap::allocate_large_object (FPO: [Non-Fpo])

01 02b5f390 791b6269 023dd040 00014c6c 00000000 mscorsvr!GCHeap::Alloc+0x15b (FPO: [Non-Fpo])

02 02b5f3a4 791b8873 00014c6c 00000000 00000000 mscorsvr!Alloc+0x3a (FPO: [Non-Fpo])

03 02b5f3c4 791b8814 02000088 0000a630 00000000 mscorsvr!FastAllocatePrimitiveArray+0x45 (FPO: [Non-Fpo])

04 02b5f444 0aa64982 02b5f488 223d1cc8 00000000 mscorsvr!JIT_NewArr1+0xbb (FPO: [Non-Fpo])

WARNING: Frame IP not in any known module. Following frames may be wrong.

05 02b5f474 791b1967 01ff2d03 00000000 184d3ba8 0xaa64982

06 02b5f47c 00000000 184d3ba8 02b5f4b8 0aa64649 mscorsvr!JIT_Stelem_Ref+0x49 (FPO: [Non-Fpo])

Executing custom commands: !clrstack;

Thread 22

ESP EIP

0x02b5f414 0x791d550d [FRAME: HelperMethodFrame]

0x02b5f44c 0x0aa64982 [DEFAULT] [hasThis] String System.Web.UI.LosFormatter.ConsumeOneToken()

0x02b5f460 0x0aa64739 [DEFAULT] [hasThis] Object System.Web.UI.LosFormatter.DeserializeValueInternal()

0x02b5f490 0x0aa64649 [DEFAULT] [hasThis] Object System.Web.UI.LosFormatter.DeserializeValueInternal()

0x02b5f4c0 0x0aa644c9 [DEFAULT] [hasThis] Object System.Web.UI.LosFormatter.DeserializeValueInternal()

0x02b5f4f0 0x0aa6450e [DEFAULT] [hasThis] Object System.Web.UI.LosFormatter.DeserializeValueInternal()

0x02b5f520 0x0aa6453c [DEFAULT] [hasThis] Object System.Web.UI.LosFormatter.DeserializeValueInternal()

0x02b5f550 0x0aa642dc [DEFAULT] [hasThis] Object System.Web.UI.LosFormatter.Deserialize(String)

0x02b5f578 0x0aa6401e [DEFAULT] [hasThis] Object System.Web.UI.Page.LoadPageStateFromPersistenceMedium()

0x02b5f5b0 0x0aa63e6b [DEFAULT] [hasThis] Void System.Web.UI.Page.LoadPageViewState()

0x02b5f5cc 0x038c74f8 [DEFAULT] [hasThis] Void System.Web.UI.Page.ProcessRequestMain()

0x02b5f610 0x038c4e67 [DEFAULT] [hasThis] Void System.Web.UI.Page.ProcessRequest()

0x02b5f64c 0x038c48d3 [DEFAULT] [hasThis] Void System.Web.UI.Page.ProcessRequest(Class System.Web.HttpContext)

0x02b5f654 0x0269e9c4 [DEFAULT] [hasThis] Void System.Web.HttpApplication/CallHandlerExecutionStep.System.Web.HttpApplication+IExecutionStep.Execute()

0x02b5f664 0x024fe6d0 [DEFAULT] [hasThis] Class System.Exception System.Web.HttpApplication.ExecuteStep(Class IExecutionStep,ByRef Boolean)

0x02b5f6ac 0x024fdd32 [DEFAULT] [hasThis] Void System.Web.HttpApplication.ResumeSteps(Class System.Exception)

0x02b5f6f4 0x024fdbfb [DEFAULT] [hasThis] Class System.IAsyncResult System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(Class System.Web.HttpContext,Class System.AsyncCallback,Object)

0x02b5f710 0x022ece3f [DEFAULT] [hasThis] Void System.Web.HttpRuntime.ProcessRequestInternal(Class System.Web.HttpWorkerRequest)

0x02b5f74c 0x022ec9f0 [DEFAULT] Void System.Web.HttpRuntime.ProcessRequest(Class System.Web.HttpWorkerRequest)

0x02b5f758 0x022e8305 [DEFAULT] [hasThis] I4 System.Web.Hosting.ISAPIRuntime.ProcessRequest(I,I4)

0x02b5f820 0x79217188 [FRAME: ContextTransitionFrame]

0x02b5f900 0x79217188 [FRAME: ComMethodFrame]

Then, I look code of method named "ConsumeOneToken()"

private string ConsumeOneToken()

{

int length = 0;

while (this._current < this._deserializationData.Length)

{

switch (this._deserializationData[this._current])

{

case ';':

case '<':

case '>':

return new string(this._builder, 0, length);

case '\\':

this._current++;

if (this._deserializationData[this._current] == 'e')

{

this._current++;

return string.Empty;

}

this._builder[length] = this._deserializationData[this._current];

length++;

break;

default:

this._builder[length] = this._deserializationData[this._current];

length++;

break;

}

this._current++;

if (length >= this._builder.Length)

{

char[] destinationArray = new char[this._builder.Length + 0x30];

Array.Copy(this._builder, destinationArray, this._builder.Length);

this._builder = destinationArray;

this._recyclable = false;

}

return new string(this._builder, 0, length);

}

and that code shocked me...

Tess, please, can you have commenting this???

re: ASP.NET Case Study: Bad perf, high memory usage and high CPU in GC - Death By ViewState

Tess — Mon, 03 Sep 2007 09:30:05 GMT

Hi .rip.

I am not really sure that I am following regarding what shocked you and what it is that should be fixed.

In this particular case it is deserializing viewstate and in doing so it has to allocate 0x00014c6c bytes (85100 bytes). The reason for this is that there are many controls on the page or some control on the page which is generating a lot of viewstate. Most likely a datagrid with a lot of rows in it or a dropdown with a lot of entries.

If you see a problem because of this, the best course of action would be to reduce the amount of viewstate generated by the page, both for the sake of mem usage/CPU but also for the sake of bandwidth, especially if you have customers potentially using dial-up or slow connections.

Tess

re: ASP.NET Case Study: Bad perf, high memory usage and high CPU in GC - Death By ViewState

.rip — Mon, 03 Sep 2007 11:25:00 GMT

Thanks for answer!!!

...developers could provide some options for buffer's initialization size ("_builder" in code), but haven't made this.

The following code has surprised me:

char[] destinationArray = new char[this._builder.Length + 0x30];

Why "+0x30" ? Why not another value?

In result, when size of ViewState (after encoding) exceeds 85000, webservice load CPU on 100% some seconds.

In fact, solution "ViewState" was made very very bad.

Fortunately, System.Web altered completely in ASP.NET 2.0

re: ASP.NET Case Study: Bad perf, high memory usage and high CPU in GC - Death By ViewState

Tess — Mon, 03 Sep 2007 12:50:06 GMT

I can't really comment on the code and why decisions were made since I didn't write it, but surely most code uses constants so why not 0x30:)

Of course you are welcome to your own opinion about how viewstate was implemented, but if you ask me I would say viewstate is a pretty nice feature, considering how much it gives you for free in terms of state management.

If you don't like viewstate as it is implemented there is always the option of turning it off and/or implementing your own:) and I do encourage you to do so in the cases where the generic implementation doesn't fit your needs. Especially when it comes to selecting which controls should or should not emit viewstate in a given page.

Everything comes at a cost (cache, viewstate, session and any other state storage) and it is just a matter of finding the right trade-off and figuring out just the right amount that makes the applications performant and usable. What I am trying to do with this blog is point out what the pitfalls are so that developers can make informed decisions regarding how to use the different features.

【收藏】本周ASP.NET英文技术文章推荐[11/26 - 12/02]

Jacky_xu — Tue, 30 Oct 2007 05:48:23 GMT

本周ASP.NET英文技术文章推荐[11/26-12/02]

提醒：若朋友们想对英文文章进行翻译，则首先需要征得原作者的同意，以免日后导致无谓的版权纷争。以下内容节选自《中华人民共和国著作权法...

Optimisation du ViewState - l'enregistrer sur le server via le SessionPageStatePersister

Atteint de Javascriptite aiguë [Cyril DURAND] — Wed, 02 Jan 2008 18:46:36 GMT

Il y a quelques temps je vous avez expliqué comment modifier la façon dont le viewstate est enregistré

How does the GC work and what are the sizes of the different generations?

If broken it is, fix it you should — Thu, 17 Apr 2008 15:14:33 GMT

During our ASP.NET debugging chat there were many questions around the GC and the different generations.

CLR Inside Out: Large Object Heap Uncovered

If broken it is, fix it you should — Mon, 09 Jun 2008 11:22:28 GMT

In many of my posts I have discussed performance issues and memory issues caused by excessive use of

ASP.NET Memory - Identifying pages with high Viewstate

If broken it is, fix it you should — Tue, 09 Sep 2008 15:56:51 GMT

From time to time we get issues with high memory and performance issues due to massive viewstate. I have

Debugging Large VIewState

PFE - Developer Notes for the Field — Thu, 23 Oct 2008 00:49:29 GMT

This week I have been working with a customer that had pretty large ViewStates that were getting pushed

Debug Diag script for troubleshooting .NET 2.0 Memory leaks

If broken it is, fix it you should — Tue, 12 May 2009 12:59:05 GMT

I have put together a quick and dirty debug diag script for troubleshooting .net memory leaks. (attached