.NET Memory usage - A restaurant analogy

re: .NET Memory usage - A restaurant analogy

DuncanS — Wed, 06 Sep 2006 16:25:44 GMT

Where do you get those memory map pictures?

re: .NET Memory usage - A restaurant analogy

Tess — Wed, 06 Sep 2006 16:47:31 GMT

I knew that if I would get one comment on this post, that would be it:)

One of my colleagues wrote a sample a long time ago that parses the output from !vadump and displays it in this way.

Unfortunately I can't share the tool for different reasons, but the output from !vadump (in windbg) looks something like this if you want to write your own parser...

0:000> !vadump
BaseAddress: 00000000
RegionSize: 00010000
State: 00010000 MEM_FREE
Protect: 00000001 PAGE_NOACCESS

BaseAddress: 00010000
RegionSize: 00001000
State: 00001000 MEM_COMMIT
Protect: 00000004 PAGE_READWRITE
Type: 00020000 MEM_PRIVATE
.........

re: .NET Memory usage - A restaurant analogy

George — Wed, 06 Sep 2006 18:21:24 GMT

Thanks Tess! That was great. But now I'm hungry . . .

But seriously: Why does Task Manager show private bytes (and why call it Virtual Memory?) and not give any option to see virtual bytes? From your article I'd think virtual bytes would be a more useful statistic. If there are 10 tables in my restaurant and they are all taken up by 10 different parties of 2 then I've maxed out my available memory - but this VM Size column would show I've only used up 20 seats. If a process has reserved a ton of space I'd like to know about it - even if no one is sitting there at the moment.

I expect part of your answer will be to use perfmon. But I *like* taskmanager because I can get to it quick in a crises, see my processes immediately, and even kill one if I have to. So it would be nice if it made a little more sense.

OT: All your blog entries are pleasure to read. How about a book?

re: .NET Memory usage - A restaurant analogy

Andy C — Thu, 07 Sep 2006 00:35:20 GMT

Best description of memory management ever! :)

re: .NET Memory usage - A restaurant analogy

ACWTeam — Thu, 07 Sep 2006 04:59:08 GMT

You'll be happy to know that on Vista private pages is being shown by default instead of working set.

Interesting Finds: September 6, 2006

Jason Haley — Thu, 07 Sep 2006 05:34:06 GMT

Link Listing - September 6, 2006

Christopher Steen — Thu, 07 Sep 2006 05:36:46 GMT

More on Atlas [Via: James Avery ] Smart Client Deployment with ClickOnce - Final Manuscript Complete!...

re: .NET Memory usage - A restaurant analogy

Tess — Thu, 07 Sep 2006 09:46:09 GMT

Awesome stuff that vista shows private pages by default, that is really great.

George: I hear ya, but it is really not as strange as you might think to call it virtual memory, after all it is committed virtual memory (and to not get endlessly long column names it is listed as just virtual memory).

I do agree with you that virtual memory is a ver important counter to look at (yes, unfortunately it is only available in perfmon), but in most memory cases the virtual/private bytes difference is not the problem, so looking at private bytes will give you a very good indication if you have a problem or not. (For example, in most asp.net apps around 1 GB private bytes, the virtual bytes will be ~1,5 GB and it is hard to get a ratio much smaller than that). So for quick stuff, look in task manager for virtual mem, and for real investigations, look in perfmon at both virtual bytes and private bytes.

The other thing is that if you look at a memory dump for example, the only thing that will help you recognize who is using the memory is by looking at the committed mem. after all, the mem that is reserved but not committed contains no data, its just uninitialized space or garbage.

Having said that, there is ways to track who owns virtual bytes that are not committed. The way to do this is to use a profiler or inject a dll into the process that hooks up to calls to virtualalloc and logs the stack anytime this is called, and then removes them when the space is deallocated.

One tool that does this is a tool that we use in support called debugdiag http://www.microsoft.com/windowsserver2003/iis/diagnostictools/default.mspx

HTH

re: .NET Memory usage - A restaurant analogy

nativecpp — Thu, 07 Sep 2006 21:02:38 GMT

Hi Tess,

Very good article. A question on your example of table:

From the pic (table), it seems that all tables contains at least one red (committed). How can the following statment be true since you can't split the reserved into two different area:

"Even though you could give fit in two 2-seat tables that wouldn't be good since they all want to sit together. "

Am I correct or I am missing something ?

Thanks

re: .NET Memory usage - A restaurant analogy

Tess — Fri, 08 Sep 2006 11:49:24 GMT

I am not sure I follow...

In the picture above we have had the following parties along with the types of tables they were assigned to

party of 1 - 2 seat table
party of 4 - 6 seat table
party of 3 - 4 seat table
party of 2 - 4 seat table
party of 4 - 4 seat table
party of 1 - 4 seat table

It would have been more economical when it comes to space for the last party of 1 to sit at a 2 seat table instead of a 4 seater, but picture it like this...

you call in for reservations, hostess: how many are you? you: not sure, i think we'll be 7... so you get a table of 8, and then some of your pals never show up, but now you are already seated at a table of 8. Similarily in the managed world you sometimes over reserve.

HTH

re: .NET Memory usage - A restaurant analogy

Petros — Fri, 08 Sep 2006 16:02:01 GMT

I am really glad that there are not any real restaurants with GC-hostess. Why? I am a slow eater and I would not like to change seats during my meal.

Interesting Finds: September 6, 2006

Jason Haley — Fri, 08 Sep 2006 18:26:42 GMT

re: .NET Memory usage - A restaurant analogy

Padma — Tue, 12 Sep 2006 09:00:38 GMT

Great blog.

I have some questions on the memory counters -

If the difference between the Private bytes and Virtual Bytes is high , does that indicate a problem? In our managed process we see that the private bytes is 250 MB whereas the virtual bytes is 500 MB. Does this mean there is some sort of fragementation?

One more question, some of the code pages of the base dlls are shared across the process right, will those also show up in the Committed memory of a process i.e. private bytes?

If working set is the pages recently used by the process , why do we sometimes see that Working Set Size (memory usage column in the taskmgr) is greater than the private bytes?

thanks , Padma

re: .NET Memory usage - A restaurant analogy

Tess — Tue, 12 Sep 2006 09:23:39 GMT

Thank you Padma,

It is fairly common that there is a difference between virtual and private bytes (up to around 500 MB or so if memory usage is high). For example at the start of the process we will have one small object segment (around 64 MB) and one large object segment (around 16 MB) per processor (for server GC). If these are not filled with data yet, virtual bytes will be a lot higher than private bytes. Similarily dlls dont fit perfectly into the chunk sizes that can be reserved etc. so I would start looking at VM fragmentation if the virtual bytes start increasing independently of private bytes or if for example you start getting out of memory exceptions at 300-400 MB. Then you have a problem of someone reserving a lot more than they need. If your app is purely managed, this is generally not something you have to worrry about since you don't have any custom components that would make unneccessary reservations. One thing that does waste virtual bytes though is having debug=true in web apps.

Private bytes are only private bytes so no shared bytes show up.

Finally, working set contains shared bytes.

Thanks,

re: .NET Memory usage - A restaurant analogy

nativecpp — Tue, 12 Sep 2006 19:36:47 GMT

I must be missing something. As you stated, blue areas are reserved space, red means committed space and white is free space. If I want to make a reservation, GC would need to use the white space to setup a table for me, right ? GC can't use the blue areas since they are reserved and can't fill in the request in some of the empty slot of the blue area ? I guess I misunderstood that you were talking about the blue area.

Thanks

re: .NET Memory usage - A restaurant analogy

Tess — Wed, 13 Sep 2006 09:35:14 GMT

Hi NativeCpp:)

You are right in that the GC would need to use the white space to set up a table. What happens is that the GC allocates its memory in chunks and committs out of it.

For example, at startup of a .net process (server GC) you get one small object heap segment (64 MB) per processor for managed objects. This is reserved and "owned" by the GC. When you do:

MyClass s = new MyClass();

rather than having to reserve a new slot on some heap, the GC uses it's reserved memory and committs memory out of it. In other words, the GC has a large blue table (reserved), and when you create a new .net object it seats you and marks it red (committ).

In the Native C++ world this would work a little differently i believe since virtual memory for C++ objects is not reserved ahead of time, but rather when the object is actually created.

The differences between the C++ world and the .net world are so large that they are almost apples and oranges. In the C++ world you manage the memory yourself (taking care of how much you need to allocate/reserve to fit an object, and taking care of when to remove stale objects), in the .NET world the GC does this for you.

The problem in the Native world when it comes to memory leaks is often forgetting to free mem, or loosing a pointer to a memory area.

In the Managed (.net) world those kind of memory leaks would rarely or ever occurr. Rather, the memory issues that occurr in the managed world are because you hold a reference to an object, keeping it alive a lot longer than you are meaning to.

I know I strayed away a little from the original question in the end but hope it helps

re: .NET Memory usage - A restaurant analogy

padma — Wed, 13 Sep 2006 13:05:15 GMT

Thanks for you response. Could you explain me the Server GC and do we get any added benifit using this?

re: .NET Memory usage - A restaurant analogy

Tess — Wed, 13 Sep 2006 15:53:50 GMT

Hi Padma,

There are a lot of articles about the GC on MSDN, but one blog that specifically deals with the GC and does so quite well is http://blogs.msdn.com/maoni/archive/2004/09/25/234273.aspx
Where the different GC flavours are described.

re: .NET Memory usage - A restaurant analogy

nativecpp — Wed, 13 Sep 2006 19:18:57 GMT

Thanks for the additional clarification. I agreed with you that native C++ and managed are totally different as far as memory is concerned. I also agreed that in managed world we are more concerned about memory rentention than leak. One of my daily activities (weekdays :-) )is to read your blog.

As for your previous articles on debugging memory issues using SOS, they are good. But I would love to have some 'real' or 'tough' cases where you need to dig deep to ioslate the problem.

Keep up the good work.

Thanks

re: .NET Memory usage - A restaurant analogy

Zeng — Mon, 02 Oct 2006 19:22:28 GMT

Test, great analogy. A question I haven't been able to find answer to is: which memory memoryLimit in machine.config is about? - If it's private bytes then is GC aware of this limit to collect memory more often when it's approaching this limit? - If it's virtual bytes then is GC aware of this limit to stop itself from over allocating when it's approaching this limit? thanks

re: .NET Memory usage - A restaurant analogy

Tess — Tue, 03 Oct 2006 10:05:32 GMT

Hi Zeng,

The memory limit in machine.config is only for ASP.NET and it is based on the amount of private bytes in the process. The GC as such is not aware of it, but ASP.NET is and will purge the cache when you get close to the limit. Allocations can still happen but once you reach the limit the process will start a recycle.

re: .NET Memory usage - A restaurant analogy

Zeng — Wed, 04 Oct 2006 18:54:52 GMT

Tess, thanks for responding. I don't know what "the cache" you referred to is, could you explain here? And if I understand you correctly there is a great chance that the process gets recycled just because GC does collect often enough especially when a big tree of nodes are waiting to be colllected, is that right? Thanks again.

re: .NET Memory usage - A restaurant analogy

Tess — Thu, 05 Oct 2006 09:27:03 GMT

The cache I am referring to is the ASP.NET cache. The memory limit is only applicable to ASP.NET applications, it does not apply to winforms apps or windows services.

If it is a winforms app or windows service there is no recycling. If the process "crashes" it won't start up again automatically. If it is an ASP.NET app it will be recycled when it reaches the memory limit. This doesn't have anything to do with how much the GC has to do.

re: .NET Memory usage - A restaurant analogy

Fredrik — Wed, 01 Nov 2006 17:16:27 GMT

Great, and entertaining, article!

I am having a situation where my application has about 70Mb in privates and >300 Mb in virtual bytes, similar to other comments here.

When looking a bit in windbg I find that MEM_IMAGE (RegionUsageImage) corresponds to almost 200Mb. At a glance this seems to be related to dlls and such that are used by the application. Is this a normal size for these objects?

Can anything be done to reduce the amount?

re: .NET Memory usage - A restaurant analogy

Tess — Thu, 02 Nov 2006 10:10:08 GMT

Hi Fredrik,

Depends on what you consider normal for your application. If it is 200MB virtual for dlls I wouldn't be too worried but you can run lm to look at the dlls that you have loaded to see if there is a lot that you dont expect. Then you can run !dumpdomain to see the assemblies for each domain and !dumpdynamicassemblies to see the dynamic assemblies. If you have a lot of those you should look into it (see previous posts on dynamic assemblies/XMLSerializer).

Also if it is an asp.net app, check that you don't have debug=true in your web.configs. But overall, 70 MB private bytes is pretty small so I don't think you have a problem with dlls based on that.

re: .NET Memory usage - A restaurant analogy

Morgan — Tue, 14 Nov 2006 16:44:58 GMT

I am using Vista RC1. It is shown as "Memory(Private Working Set)" in task manager, but the value seems have no relationship to private bytes, virutal bytes or working set in perfmon at all.

Perhaps it is correct in RTM.

re: .NET Memory usage - A restaurant analogy

Tehnoon — Mon, 04 Dec 2006 10:42:09 GMT

That is absolutely amazing dude! Excellent excellent stuff!

re: .NET Memory usage - A restaurant analogy

Jaaloor — Wed, 31 Jan 2007 10:47:22 GMT

I have a question. We encoutner troubles (you'll see it's normal we do when I'll have finish explaning what we are trying to do ;) with a .Net appliation. We have an application that stores document for users. We upload documents into the http request, to do so we modified the maxRequestLength parameter in the web.config file of the application. The problem is that we want to allow user to upload documents up to 100Mb (yes I know but I am asked for a miracle !). the result is that sometimes we can add such big documents and sometimes we can't. The server as loads of physical memory.

The thing is that the memory usage for the application never exceed 500Mb or something like that. I fear that when the upload fails it is because it starts to store the request in the memory and at some point don't find any free contiguous memory slot to keep on loading the request. I would expect that it reserves all the memory when the request arrives.

If anyone has any idea regarding this issue I would be very thankfull !

re: .NET Memory usage - A restaurant analogy

Tess — Wed, 31 Jan 2007 11:09:05 GMT

You might want to check out

http://blogs.msdn.com/johan/archive/2006/11/15/are-you-getting-outofmemoryexceptions-when-uploading-large-files.aspx

re: .NET Memory usage - A restaurant analogy

Jaaloor — Wed, 31 Jan 2007 11:25:18 GMT

thanks for your quick answer !

In fact that is exactly What I try to make understand to my customer but still they first want me to do a miracle ...

re: .NET Memory usage - A restaurant analogy

Jaaloor — Wed, 31 Jan 2007 20:19:38 GMT

thanks for your help.

In fact that I try to say to my customer for the whole week :P one more evidence helped me to bring him to reality :) now we're going towards a realistic solution :P

Administración de la memoria en Windows y .NET

la visión de un ingeniero de campo — Wed, 07 Feb 2007 07:16:03 GMT

Veremos de firma simple y didáctica como funciona la asignación y administración de memoria de Windows y .net.

re: .NET Memory usage - A restaurant analogy

Alex — Sun, 01 Apr 2007 15:03:42 GMT

Tess I think your great (I had a fortune of getting help from you on number of support cases and you are definetly one of the best MS support engineers I worked with).

Does microsoft (perhaps internally) has a utility that can analyze some of this stuff automaticly? If you have the dump then I am sure alot of these problems can be diagnosed automaticly. Or am I over simplifying here?

Alex.

re: .NET Memory usage - A restaurant analogy

Tess — Mon, 02 Apr 2007 09:14:04 GMT

Hi Alex,

First off, thank you very much:) that makes me blush:)

We are working on scripts for automating some parts of .net debugging. You may be familiar with a tool for automation of native debugging called debug diagnostics 1.1. In essence we are looking into what we can do as far as writing scripts for that for .net debugging, but there are some parts, especially memory wise that are a bit hard. For example figuring out which objects are interesting to look at based on the !dumpheap -stat output and then determining what roots are interesting since the job is based on a lot of trial and error.

It is fairly easy to write scripts on your own for debug diag (vbscript files) and a good start is to write one that prints out output from !dumpheap -stat, !dumpheap -min 85000, !eeheap -gc, ~* e !clrstack, !threads, !dumpdomain, !syncblk and ~* kb

If you just create that "simple" script it takes you a long way into seeing what is going on in most cases, and writing the script is as easy as writing

Manager.Write g_Debugger.Execute("!dumpheap -stat")

in one of the scripts in the debugdiag scripts folder (of course you have to make sure to load sos first).

Before we have anything that is usable in a more general sense it might be a while, but as soon as we do, I will post it on the blog.

re: .NET Memory usage - A restaurant analogy

Jes — Mon, 21 May 2007 17:23:44 GMT

Hi Tess,

Great stuff!

I am trying to figure out how much RAM our .NET app. is actually using and used your recipe above monitoring with Private and Virtual bytes with PerfMon. This seems to make sense, BUT:

1) Why is page file constantly growing, when their is sufficient with RAM left? It grows with app. the same size as RAM being used. Has it something to do with Virtual memory always requires disk??? (I am blank :-)))

2) Why is a high RAM usage "maintained" even after data has been discarded? (first I complete heavy search, then complete less heavy search...dataset is nulled each time) I would expect it to go down immediately.

The hierarchy in flat memory: Heap and Stack

Li Xiong — Mon, 11 Jun 2007 18:59:05 GMT

The hierarchy in flat memory: Heap and Stack This section discusses Heap, related heap corruption/memory

Generational GC (Garbage Collector) - A post-it analogy

If broken it is, fix it you should — Tue, 26 Feb 2008 18:08:47 GMT

I was working through the High CPU Lab Review which is basically caused by high CPU in GC. To understand

re: .NET Memory usage - A restaurant analogy

Amol — Wed, 09 Jul 2008 08:22:47 GMT

Tess, that's a wonderful way to explain memory management. I had a query about the following statement though.

"When you allocate an object (whether it is .net or non-.net) you typically follow a two step process. You reserve the memory and then you commit space inside your reservation."

In context of .Net memory management, when is that we reserve the memory and when is it commited. Can you please elaborate?

re: .NET Memory usage - A restaurant analogy

Tess — Mon, 04 Aug 2008 22:39:31 GMT

memory on the .net gc heaps are reserved in segments so everytime you fill up a segment and need more space you would reserve a segment which might be for example 16 MB, 32 MB, 64 MB or even larger depending on framework version, GC mode, OS architecture etc.

Either way, you reserve the 64 MB for example but you wouldn't commit memory inside them until you actually allocate a new .net object. Now if that .net object was just 2k then you would have ~64 MB reserved and 2 k committed... if you allocate another 3k object you would then have 64 mb reserved and 5k committed etc. etc.

re: .NET Memory usage - A restaurant analogy

Danp129 — Thu, 14 Aug 2008 10:57:17 GMT

Thanks for writing this.

For everyone that doesn't like Task Manager, try Process Explorer, it's free and it shows all the memory info you want and has tons of features.

http://technet.microsoft.com/en-us/sysinternals/0e18b180-9b7a-4c49-8120-c47c5a693683.aspx

I recommend the full suite or atleast looking at each app in the suite and choosing what you like.

re: .NET Memory usage - A restaurant analogy

Murali — Fri, 21 Nov 2008 01:35:16 GMT

Hi Tess,

Its an excellent article on memory management. Is there any way to identify the memory being consumed by DLL's used.

For ex:

A MOSS application has many third party webparts and controls and http handlers. How can we isolate which webpart or which control is taking more memory.

Because I have observed that application stops serving some of the HTTP handlers until we do a app-pool recycle. Then immediately that handler works and serves the requests and again stops after some time...

re: .NET Memory usage - A restaurant analogy

Tess — Fri, 21 Nov 2008 01:50:04 GMT

There isnt really a way to record memory allocated by dll (assuming you mean memory allocated by the code in that particular assembly)

You would have to do a normal memory investigation, check out some of the labs around memory (on the right hand bar) to see how you can do memory investigations...

re: .NET Memory usage - A restaurant analogy

Jonathan — Wed, 03 Dec 2008 20:21:32 GMT

I stumbled upon your article doing a Google search for why Task Manager didn't reflect the correct memory usage. When adding the memory usage for 6 processes it would be up to 8GB and the server only has 3GB! Your article really cleared things up for me by explaining the difference between private bytes & virtual bytes. I'm still not sure how those processes can amount to 8GB but thanks for writing this article!

re: .NET Memory usage - A restaurant analogy

Tess — Thu, 04 Dec 2008 01:56:20 GMT

while you only have 3 GB of RAM that doesnt mean that the combined memory usage for your processes can't be more. Any memory that doesn't fit in RAM will be swapped out to disk in page files

Linki do prezentacji o zarządzaniu obiektami

.neting in the free world — Tue, 16 Dec 2008 01:40:12 GMT

Linki, które posłużyły mi przy tworzeniu prezentacji, z których czerpałem wiedzę, nakładałem ją na to

re: .NET Memory usage - A restaurant analogy

Hemal — Tue, 10 Mar 2009 13:12:39 GMT

Hi,

I have a problem...

Please help me out.. i m using asp.net 2005 and my application is to uplaod multiple images on the server, but it gets crash down when multiple user tries to uplaod images. the error is like this

aspnet_wp.exe (PID: 3040) was recycled because memory consumption exceeded the 609 MB (60 percent of available RAM).

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Means something related to memory, server is having 1GB RAM..

Please help me out, i m stuck.............

Thanks in advance

re: .NET Memory usage - A restaurant analogy

Tess — Tue, 10 Mar 2009 13:22:47 GMT

you might want to read through

http://blogs.msdn.com/johan/archive/tags/Upload/default.aspx

Regarding the 60%, this is the default setting for 1.1 in machine.config, you can change it if you dont want it to recycle at 60% of RAM size, but you will probably get out of memory exceptions if your images are very large

re: .NET Memory usage - A restaurant analogy

Josh — Thu, 12 Mar 2009 20:45:02 GMT

I was just reading through this post and marveling at the simplicity and elegance of the explanation, after which I happened to scroll back up to the top. It took me a couple of moments to realize that's probably a picture of the writer up at the top of the page.

So you have someone walking around with that kind of brain power that also looks like that?! Tess is either a mythical avatar that can't be found in nature or she won the gene pool lottery.

re: .NET Memory usage - A restaurant analogy

Amjad — Thu, 19 Mar 2009 16:39:45 GMT

Good article, here i want to relate it with my scenario.

I initialize all the object, which will be needed, in advance and put them in a map and i return the clone of object on demand.

I want you to shed light on this approach and what to know is there ever a chance of memory leaks in this scenario. Thanks

re: .NET Memory usage - A restaurant analogy

JC — Thu, 02 Jul 2009 15:03:14 GMT

Really great article.

I have a query may be you can help.

In our application we create big arrays of the size

float[] abc = new float[65000] many such arrays are created. As many as 512 or 1024. We get outofmemory exception frequently.

The problem is even if we set these arrays to null memory is not released.

We doubt that GC is not reclaiming the memory.

We did memory profiling using some tools then the tools shows that the arrays are removed.

Can you suggest something for this problem.

re: .NET Memory usage - A restaurant analogy

Tess — Thu, 02 Jul 2009 15:30:11 GMT

JC,

Because of the size these will end up on the large object heap which is not garbage collected as frequently as small objects.

Depending on framework version etc. and allocation pattern you may also be causing some significant fragmentation on the large object heap here.

The GC should collect them if they are ready to be collected the next time a full GC is initiated.

I am thinking the reason for your OOMs is probably that you allocate a lot of these at the same time (i.e. their lifetimes intersect) and my best advice to you would be to make them slightly smaller so that they dont end up on the large object heap. Monitor the perf counter for LOH to see how much you allocate on the LOH.

re: .NET Memory usage - A restaurant analogy

Sandeep — Thu, 05 Nov 2009 23:23:01 GMT

Hi Tess

Great article! But the memory map pictures are not loading on this page...can this be fixed?

Thanks

re: .NET Memory usage - A restaurant analogy

Sandeep — Fri, 06 Nov 2009 15:05:33 GMT

Hi Tess

Great article....but the memory map pics are not loading on this page....can this be fixed?

re: .NET Memory usage - A restaurant analogy

Tess — Fri, 06 Nov 2009 18:44:13 GMT

sandeep, are you talking about pictures? I can see the pics fine, do you get an error of some kind?

re: .NET Memory usage - A restaurant analogy

sandeeparora — Fri, 06 Nov 2009 21:25:54 GMT

For me, either in IE or Firefox, it just shows a [x] in those places where images should be loaded...I would appreciate if u can mail them to me at this Id : sandeeparora_1704@yahoo.com

Thanks in advance

Sandeep