I found myself writing a very long response to a recent comment to this blog entry, so I decided to make it into a blog entry and link the two.

Question:

Thanks for the info. Sorry i was not clear.

I am recycling the memory based upon the virtual memory size. I presumed that by virtual memory it means memory stored on disk in the page file. Is this not the case?

There is still physical ram free on the machine when i see the out of memory problems leading me to suspect either a memory leak or memory fragmentation. I see the virtual bytes counter rise quickly to 1500mb (2-3 times a day) whereas the physical Ram used by the w3wp process is only a few hundred MB with plenty of RAM left to spare.

I'm not sure why the virtual memory grows so high when there is enough Physical RAM. I have tried using DebugDiag but the report generated is too high a level for me to understand without more documentation. An example from the DebugDiag report on the w3wp process when it had a high virtual memory usage:

Virtual Memory Summary
Size of largest free VM block 75.21 MBytes
Free memory fragmentation 88.13%
Free Memory 633.53 MBytes (30.93% of Total Memory) 
Reserved Memory 1.03 GBytes (51.29% of Total Memory) 
Committed Memory 364.06 MBytes (17.78% of Total Memory) 
Total Memory 2.00 GBytes 
Largest free block at 0x00000000`685ba000

This is the heap I identified as using the most memory

Heap Name
  msvcrt!_crtheap 
Heap Description 
  This heap is used by msvcrt 
Reserved memory 868.25 MBytes 
Committed memory 89.16 MBytes (10.27% of reserved) 
Uncommitted memory 779.10 MBytes (89.73% of reserved) 
Number of heap segments 12 segments 
Number of uncommitted ranges 8207 range(s) 
Size of largest uncommitted range 32.20 MBytes 
Calculated heap fragmentation 95.87% 
Top Allocation - Allocation Size - 1048
29.42 MBytes 29435 allocation(s)

There are 12 heap segments and nearly all have extremely high fragmentation and they have allocated a lot of reserved memory. The commited size in each segment is very low so i'm unsure why that is the case and where i go from here to debug the application further.

This has been driving me nuts for months now and there isn't anyone i know to ask other than you experts who write this stuff...

Going back to your article you say that i should run the ASP on iis in a 64 bit process and that it will automatically create a seperate dll host for all the 32 bit com objects.

Do i need to set the flag Enable32BitAppOnWin64 for this to work?

How do i register the 32 bit components. Is it a case of simply using regsvr32?

Answer:

Ah, I see where things are going awry...

About Virtual Memory

Virtual Memory is not the same as memory stored on disk in the page file.

Virtual Memory is a concept of indirection introduced in memory addressing whereby application see a contiguous "virtual memory" address space, but the OS transparently fulfills that memory address space with EITHER physical memory (i.e. RAM) or something else like disk memory (i.e. pagefile).

From the application's perspective, it is simply using "memory". However, the OS knows whether that "memory" is real physical memory or has to be swapped in from the pagefile on disk into real memory. It is this swapping operation which makes virtual memory that is in the page file slow.

If you want more information on the subject, I suggest reading books like "Inside Windows 2000" or "Microsoft Windows Internals" by Solomon and Russinovich, or classic books like "Computer Architectures: A Quantitative Approach". I know that it's probably way more information and detail than you are asking for, but those are the authoritative sources that I go with...

Memory Fragmentation vs Memory Leak

The classic signs of Memory Fragmentation is if you have lots of physical RAM free, low committed memory, yet seeing Out Of Memory errors.

How to distinguish this from Memory leak? Well, Memory leak would consume your physical RAM such that very little is free, virtual memory would be maxed out, and you see Out Of Memory errors. When you see the error depends on when your Virtual Memory depletes - a pagefile simply delays the inevitable and eventually grinds the system to a halt.

Your issue looks like memory fragmentation.

About Memory Fragmentation

Memory fragmentation is really an issue caused by user-application not designed for long-term usage such as on a server. The best way to fix the issue is to architect the application correctly with regards to memory allocation.

When applications just malloc/free memory all the time, over time this will naturally fragment memory. How does this happen? Well, assume the application does the following sequence of memory operations (pardon the ASCII art attempt to visualize what's going on in the virtual memory address space...):

  1. Allocates 4K of memory |--4K--|
  2. Allocates 4K of memory |--4K--|--4K--|
  3. Deallocates the first 4K of memory |--??--|--4K--|
  4. Allocates 8K of memory |--??--|--4K--|--8K--|

Since applications assume memory address is contiguous, that 8K allocation CANNOT use any of the 4K of deallocated (??) memory. That 4K is technically "free memory", but if the application needs >4K of memory, it is not usable by it. Now, imagine this happening all over the virtual memory address space to the point that no big chunk of CONTIGUOUS memory remains, and then you ask for a big block of memory... this is when you get Out of Memory even though enough fragmented free memory exists.

Here's a simplistic "worst case" example: assuming you manage to allocate the first byte of every virtual memory page (let's say the page size is 4096 bytes) and the virtual memory address space size is 4GB, you will see only 1MB of physical RAM used but all of the virtual memory used... and you get Out of Memory on the next allocation. In other words, over 99.99% memory is "free" yet you get "Out of Memory". Crazy, right? But that's memory fragmentation for you...

Now, software designed for server-side usage pre-create and cache memory blocks for runtime usage to avoid hitting the heap and eventually causing memory fragmentation. For example, IIS is architected to do this; it does not call malloc/free at runtime to handle requests... though we cannot cover user applications...

On Running 32bit COM objects on 64bit Windows

You do not need to change Enable32BitAppOnWin64 to run 32bit COM components. That property just makes IIS launch 32bit w3wp.exe to execute user code. 64bit process can use 32bit COM components just fine... just outside of its process instead of in-process, and this happens transiently. Of course, this automatic alteration may/not work, depending on how your COM component functions.

Also, it is easiest to just use the 64bit version of REGSVR32.EXE (by default). Please do not accidentally run a 32bit command console window and use the 32bit version of REGSVR32. REGSVR32 tries to put the ProgId into the necessary places depending on the bitness of the DLL such that 32bit processes load 32bit DLLs and 64bit processes load 64bit DLLs

Of course, whether this works or not depends on if the 32bit COM component actually works/registers correctly on 64bit Windows, and that is best determined by the people supporting that component.

//David