Slava Oks's WebLog : Memory Management

Q and A: Ratio between AWE Allocated and VM Committed/Reserved in DBCC MEMORYSTATUS output

slavao — Wed, 03 Jan 2007 04:02:00 GMT

Q: What kind of ratio should you expect to see between AWE Allocated and VM Committed/Reserved? We are running 8GB with a cap of 7GB in the max server memory and yet the VM Committed is around 100MB. I was first shocked to see that the sqlservice in task...(read more)

Q & A: I am running on 64 bit and all of my tasks are waiting on SOS_RESERVEDMEMBLOCKLIST, What is going on?

slavao — Tue, 14 Nov 2006 21:29:00 GMT

This wait type represents waiting on multi page allocations inside of SQLOS's memory manager. As you remember from my description of memory manager's working internals, it has two major allocators single page allocator, SPA and multi page allocator, MPA....(read more)

Be Aware: 4GB of VAS under WOW, does it really worth it?

slavao — Mon, 13 Mar 2006 03:12:00 GMT

By now you have heard a lot about the fact that under WOW a 32 bit process can get 4GB of VAS. I agree this is great, but I would like you to be aware of something that might hit you really hard. As it turns out enabling 4GB VAS can be problematic due...(read more)

Be Aware: VirtualAlloc with MEM_RESERVE can fail even if you have plenty of VAS inside of the process

slavao — Fri, 03 Jun 2005 19:04:00 GMT

Many developers assume that if you have plenty of free VAS in your process VirtualAlloc (http://msdn.microsoft.com/library/default.asp?url=/library/en-us/memory/base/virtualalloc.asp) and VirtualAllocEx (http://msdn.microsoft.com/library/default.asp?url=/library/en-us/memory/base/virtualallocex.asp) calls with MEM_RESERVE parameter can't fail. This is not actually true. Remember that when allocating VAS region OS creates VAD (Virtual Address Descriptor). VAD manages VAS region, for more info read http://blogs.msdn.com/slavao/archive/2005/01/27/361678.aspx. When system is low on physical memory which includes RAM and swap file, VAD's allocation might fail. The failure will cause VirtualAlloc* call to fail. It means that if system low on physical memory VirtualAlloc* could fail. This becomes very important on 64 bit platforms. When running on 64 bit, developers might assume that reservation never fails - which is wrong.

Hope it was useful.

Be Aware: Using AWE, locked pages in memory, on 64 bit

slavao — Fri, 29 Apr 2005 15:23:00 GMT

We have already talked about Windows AWE mechanism on 32 bit and how SQL Server utilizes it. Today I would like to go over AWE & related mechanism on 64 bit platforms.

To some people it comes as a surprise that AWE mechanism is still present and actually could be useful on 64 bit platforms. As you remember the mechanism consists of two parts allocating physical memory and mapping it to the given process's VAS. The advantage of allocation mechanism is that once physical memory is allocated operating system can't reclaim it until either the process is terminated or the process frees memory back to the OS. This feature allows an application to control and even avoid paging altogether. Advantage of mapping/unmapping mechanism is that the same physical page could be mapped into different VAS's regions. As you imaging unmapping is not necessary on 64 bit platforms since we have enough VAS to accommodate all of existing physical memory.

From Operating System theory, OS implements a page table entry, PTE, to describe a mapping of a page in VAS to physical page. Internally physical page is described by page frame number, PFN. Given PFN one can derive complete information about physical page it represents. For example PFN shows to which NUMA node the particular page belongs. OS has a database, collection of PFNs that it manages. If page in VAS is committed, it has PTE which might or might not point to given PFN. Conceptually, page that PTE represents can be either in memory or not, for example swapped out to disk. In the former case it is bound to a given PFN and in latter it is not. In its turn, once a physical page is bound to page in VAS, its PFN points back to PTE.

When OS commits, frees, pages out/in a given PTE or needs to derive some information about it, for example NUMA residency, it has to acquire process's working set lock - to guarantee stability of PTE to PFN binding. This lock is a rather expensive and might hurt scalability of the process. Latter versions of Windows made this lock as light as possible but avoiding still will benefit application's scalability..

When allocating physical pages utilizing AWE mechanism we are given a set of PFN entries directly from PFN database - remember that you should not manipulate or modify set of entries you get back nor can you rely on values you get back. OS is required to take a PFN database lock when allocating PFN entries. Using AWE map mechanism you can map allocated PFN entries to the process's VAS. When mapping occurs PTEs are allocated, bound to PFNs and marked as locked. In this case OS needs to acquire process's working set lock only ones. When mapping regular pages, OS does it on demand and hence will have to acquire both working set and PFN database lock for every page. Since pages are locked in memory, OS will ignore these PTEs during paging process.

On 64 bit platforms it is better to refer to such pages as locked pages - please don't confuse them with pages locked through VirtualLock API. As described above locked pages have two major properties - they are not considered for paging by OS and during allocation they acquire both working set and PFN database lock only ones.

The first property has implicit implication on high end hardware such as NUMA. It provides explicit memory residency. Remember that OS commits a page on demand. To allocate physical memory, it will use a node on which a thread touching memory is running. Latter on, the page can be swapped out by OS. Next time it will be brought up into memory, OS will again allocate physical page from the node a thread touching memory is running on. In this case a node could be completely different from original one. Such behavior makes hard for applications to rely on page's NUMA residency. Locked pages solve this problem by removing themselves from paging altogether. Moreover Windows 2003 SP1 introduced a new API - QueryWorkingSetEx. It allows to query extended information about PTE's PFN. In order to find out real page residency you should use this API. When pages are locked you need to it only ones. Otherwise you will have to do it periodically since residency of the page can actually change.

The second property - taking both working set and PFN's database lock only ones enables applications to perform faster and better scalable ramp up.

On NUMA SQL Server' Buffer Pool marks each allocated page with its node residency. It leverages QueryWorkingSetEx to accomplish it. Once page is allocated it calls the API to find out page residency. It does it only once. Therefore enabling locked pages for SQL server on 64 bit platform will improve SQL Server ramp up time and will improve performance & scalability over longer period of time. When running SQL Server with locked pages enabled you shouldn't be worried about overall system performance due to memory starvation - SQL Server participates in OS's paging mechanism by listening on OS's memory notification API's and shrinks its working set accordingly.

Let us summarize now - on 64 bit platform, locked pages, awe mechanism, enable better application's scalability and performance both during ramp up time and over long period of time. However, keep in mind that an application is still required to implement a way of responding to memory pressure to avoid starving the whole system for memory.

Let me know your comments!

Look at Memory Leaks - Classify and Identify

slavao — Fri, 04 Feb 2005 20:16:00 GMT

In my previous post I talked about type of Memory Pressures. Today I would like to talk about related subject – Memory Leaks. Before you continue reading please make sure that you are familiar with how memory is managed on Windows.

Memory leaks are bugs that happen very often. I think it is very important to know how to approach the issue. Once you learned the process, these bugs are not as evil as everyone things.

When leaks occur, the first thing developer needs to do is to classify them. How many times you have heard people are just saying the application failed or became unresponsive due to memory leak? A phrase memory leak could mean leak of several memory resources. Remember in my previous posts I talked about VAS, physical memory, page file and etc. An application can leak any of these resources. This means that memory leak is incomplete phrase that doesn’t have much meaning. One needs to be very specific when referring to a leak.

Leak Classification is very important. The table below represents memory leak type and tools that could be used to classify them. Keep in mind that there could be more tools and more leak types. I just enumerated the most common types that I dealt with in the past.

Memory Resource/Tool	Task Manager	PerfMon	WinDbg	WinDbg – Local Kernel Debugger	VADUMP	SQL Server
VAS		Process/Virtual Bytes	!vadump !heap	!vad	X
Physical Memory	Virtual Memory	Process/Private Bytes	!vadump !heap	!memusage !vm	X
Page File	Commit Charge	Memory/Committed Bytes		!memusage !vm	X
AWE						Buffer Manager/Total pages
Handle	Handles	Objects	!handle	!handle

Once leak is classified, it needs to be identified. Identification of a leak involves a tracking of leaking stack. Once stack is known leak is identified. Keep in mind in many cases it is not sufficient to identify file line and number. Below table represents tools that could be used to identify a leak:

Memory Resource/Tool	umdh/pageheap	LeakDiag	WinDbg	WinDbg – Local Kernel Debugger
VAS		Virtual Allocator	!heap -l
Physical Memory / Page File	Windows Heaps	Windows Heaps CRT Heap COM Heap MDAC Heap	!vadump !heap	!memusage
AWE
Handle			Break points on CreateEvent, CreateMutex, … CloseHandle	!htrace

Usually classifying a leak is not hard and should be done very fast. You should be able to teach your testers, QA personal, to do it. Moreover the classsification process could be automated. Identifying a leak is usually harder and more time consuming process. It might involve multiple restarts, use of different tools and etc... Knowing how a tool works and being comfortable with it is the key. Getting there does require experience. Once you are there, dealing with memory leaks becomes easier process. The only problem is that there will be always a leak that your tools can’t handleJ. This is the case when you sit down and write your own LeakDiag.

There are several commercially available tools enabling you to classify and identify leaks. I haven’t looked at them for long time but from my past experience they don’t work as well as the ones I outlined in here. Before you decide to buy one make sure you understand how the tool works. If tool is capable of handling leaks only at process shutdown, it has very low value. Moreover, If you work on high-end server and have your own memory manager, most of those tools are not applicable.

I will be glad to hear your comments.

Enjoy weekend!

Memory Pressure - Classified

slavao — Tue, 01 Feb 2005 17:11:00 GMT

So far I touched on subjects that have been widely discussed in the industry. Today I would like to talk about a subject that you would hardly find information about: memory pressure.. On a surface the subject appears simple but in reality this is not the case.

There are two types of memory pressure a process can be exposed to external and internal. To maximize its performance and reliability a process might want to react to both of them. External memory pressure might cause a process and whole system go into paging . Internal memory pressure might cause OOM conditions and eventual process's crash.

External memory pressure is controlled by Windows, operating system. There are two types of external memory pressure such as physical dynamic memory pressure and physical “static” memory pressure. The latter type happens when a system runs out of page file. This type of memory pressure might drive the whole system into OOM condition. You might have seen those pop ups in the right corner indicating that system runs low on virtual memory. In order to detect this type of pressure one needs to monitor the size of page file. Usually applications don’t do it.

The external dynamic memory pressure rises when Windows runs low on free RAM and about to start trimming existing working sets on the box, i.e paging. A process can monitor this type of pressure by leveraging memory resource notification API described here http://msdn.microsoft.com/library/default.asp?url=/library/en-us/memory/base/querymemoryresourcenotification.asp. An application can have dedicated thread that listens on memory resource notifications. Keep in mind that these notificaitons are global, i.e. they are shared by all processes. There are two types of memory resource notifications that a thread can wait on: memory high and memory low. Before Windows starts paging it will turn on low memory resource notificaion. Applications’ threads that waiting on such notification will be waken up and given opportunity to shrink process’s memory usage before OS comes into the picture. This is very useful for highend services that have better idea than the operating system about their memory usage and what needs to be shrunk. Once memory goes back to normal Windows unsets the low memory resource notification. As you might expect, when Windows thinks there is a plenty of memory on the box, it turns on memory high resource notification. If both of the memory resource notifications are not set it means that system is in stable state and processes shouldn’t either grow or shrink.

There are two types of internal memory pressure such as physical memory pressure and Virtual Address Space, VAS, pressure. Depending on its memory manager there are several ways for a process to get into physical internal memory pressure. For example external pressure might cause a process to shrink. This might trigger process’s releasing memory which in its turn will trigger internal memory pressure. The other possibility to get into this type of pressure is if an administrator sets memory limits for a process. Once max is reached process will get into internal memory pressure. Usually application recovers from this type of pressure by shrinking internal caches and pools back to its memory manger. In cases when there is no external memory pressure there is no reason to free this memory back to operating system.

VAS pressure is the most difficult one to detect and react to. VAS pressure could happen due to two reasons. The first reason is VAS fragmentation. It happens when a process might have plenty of VAS regions but there is no a VAS region of a given size available. Currently there is no easy way to detect largest free VAS region. One could try to allocate a VAS region of given size to identify VAS pressure state. Be careful though, periodic attempt to allocate a VAS region of large sizes, say 4MB, might cause VAS fragmentation. This will happen if some component in the process keeps on allocating and caching VAS regions of smaller sizes, for example threads. The second reason for VAS pressure is the whole VAS could be consumed. In this case any VAS allocation fails. High-end servers have to be able to deal with VAS pressure especially on 32 bit platforms. Not recovering from VAS pressure might cause first process's slowing down and then terminating. When an application detects VAS pressure it could react to it the same way as to internal physical pressure by shrinking caches and pools. In addition a process might decide to shrink thread pools, remove shared memory regions, unload dlls and etc…

To correctly handle all types of memory pressure you will need to build special infrastructure. As it turns out this type of infrastructure is not simple. Just consider different states your process can be in at the same time. For example Windows might indicate that there is plenty of external RAM, enabling your process can grow, but at the same time your process can hit internal physical or VAS pressure.

There are several implementation caveats that you need be aware of when implementing such infrastructure. If your process slow enough to react to external pressure Windows will page your process out. Then it will turn off low memory resource. Once it seesm that there is plenty of memory it will turn on high memory resource. In this case you might see that high is on and start allocating more memory even though your process is paged out. This might cause your process to page against itself. It seems that when deciding to grow you need to take your working set into account, please remember neither AWE pages nor large pages are part of working set so you have to be really careful. The other caveat is when running low on paging file, Windows won’t turn on low memory resource even though it is about to return OOM for next memory requests. In addition keep in mind that your well behaved application can be affected by a bad one that doesn’t care about memory state on the box at all.

Understanding memory pressure should really help you when we will be discussing SQL Server memory manger. Moreover having this knowledge should help you to administrate SQL Server and other applications sharing a box

I will be glad to hear your comments J

Have a nice day!

A look at Windows Virtual Memory mechanisms (continuation of "A look at Virtual Address Space - VAS")

slavao — Sun, 30 Jan 2005 01:00:00 GMT

As I promised last time here comes next post on memory J. Remember, my eventual goal is to reveal how memory management works in SQL Server but for you to really appreciate it, I think, you do need to get good feeling on how Windows manages memory. Understanding details is great however at this point I want you to understand the concepts! In some cases I on purpose skip the details because right now they are not important and many of them can be changing from one release to another. So lets continue!

In my previous post I mentioned that VAS regions could be bound to physical memory right a way or latter on using VirtualAlloc API. As most of the operating system do Windows binds physical memory to the VAS on demand, the first time page in a region is touched. (The story is a bit different when you are running without paging file, then VAS page is bound to physical page right a way) The binding happens only one page at a time. When memory first touched hardware generates exception, called page fault. The exception is handled by Windows. The OS verifies if current VAS region is committed by looking into the VAD structure representing this region. If region is committed and it is accessed for the first time OS will find a physical page in RAM it can use,(Keep in mind that this page will be zeroed out before given away due to security reasons) Finally it binds VAS region to the page by filling appropriate data structures and loads this information into CPU then execution continues right at the point where page fault occurred.

Due to shortage of RAM Windows might decide to take a way physical page from a process. Using a reclaiming policy the OS finds a page that it will free. Then OS makes sure that page file contains up to date image of the page on disk by spilling page to paging file if necessary. Once the page is on disk, OS can fix up all required data structures so that next time the page is touched it will know where to find it. When finished, the physical page is zeroed out and then put on the free list to be used for subsequent memory requests.

As I said previously a CPU generates exception, a page fault, when it can’t find a physical page corresponding to a virtual address it tries to access. An application can generate page fault by two ways first when it touches VAS just committed region for the first time and second if the OS previously paged out given page to disk. (when application touches a VAS page that hasn’t been committed yet Windows will generate exception). The type of fault when page needs to be brought from disk is called hard page fault. If application touches just committed VAS region for the first time it will generate a a demand zero page fault. There is also another type of page fault – soft page fault. Soft page fault occurs when Windows can still find the page in physical memory and serve it without bringing the page from disk.

Using Windows virtual memory mechanism the same physical pages can be mapped to different VASes at different places and multiple times. Physical pages that can only be mapped to single VAS are called private, because they can’t be shared across different VASes. Physical pages that are mapped to different VASes at the same time are called shared physical pages.

All VAS regions committed using VirtualAlloc* interfaces bound to physical pages that can’t be shared across different VASes. One can think about them as private physical pages. A sum of all private physical pages, in RAM and on disk is called Private Bytes, using perfmon terminology or VM Size according to Task Manager’s terminology. A sum of all private physical pages that are only in RAM is called working set, or Memory Usage according to Task Manager.

As I mention before, depending on physical memory demands Windows might remove physical pages from process’s working set. This activity is usually referred as paging. The OS provides a way to avoid paging of a given VAS regions. It provides mechanism to “lock” VAS regions in physical RAM. As you would expect an application that tries to lock its VAS regions might destabilized the behavior of the whole system. To mitigate this behavior Windows has “Lock pages in memory” privilege turned off by default so that only applications that a specifically given this permission by administrator can lock pages in memory. In addition the OS still might page out the whole process’s working set if needed.

Even though the size of x86 registers is 32 bit the platform enables an OS to manipulate with 64GB of RAM using Physical Address Extensions, PAE. On Windows PAE can be turned on from boot.ini file. If PAE is not turned on, Windows will only be able to manipulate with up to 4GB of RAM even though box might have more of RAM installed.

Some user applications require more VAS than 2GB. Windows have ability to configure VAS of user application up to 3GB. This feature has significant drawback. It limits amount of VAS available to kernel down to 1GB. Increasing size of user VAS can be done by adding /3GB switch to boot.ini file. Once modification is complete system will require reboot.

Limiting kernel VAS to 1GB affects the whole box not only the application that needs larger size VAS. For example ones 3GB switch is turned on amount of RAM that the OS can support, if PAE is enabled, drops down from 64GB to16GB. 3GB switch affect all kernel components including all drivers. Having 3GB switch turned on can cause effects from drop in performance and memory allocation failures to system stalls. My suggestion is to avoid usage of 3GB switch unless you really really need it.

As I described, on x86 platforms VAS is limited to 2GB or 3GB depending on configuration in boot.ini file. On AMD64, 32 bit application can have up to 4GB. Such small VAS could be a significant drawback for high end servers manipulating with gigabytes and terabytes of data, think about SQL Server. I also said that PAE switch enables Windows to manipulate with up to 64GB of RAM. But how one process does access so much memory? In order to enable single process to manipulate with amount of RAM bigger than size of VAS, Windows provides Address Window Extension, AWE, mechanism (Remember your DOS yearsJ). Be careful many authors use term AWE memory. There is no such thing as AWE memory, one can’t go and buy AWE memory at a store J. . Since there is no AWE memory, there is no either low AWE nor high AWE memory. There is AWE mechanism that enables one to manipulate with amounts of RAM larger than VAS! The principle is simple. Using AWE API one can allocate physical memory, then using VirtualAlloc API allocate VAS regions and then using AWE mechanism bind/ubind VAS regions and physical memory. As with locked pages in order for an application to use AWE mechanism it needs to have Locked pages in memory privilege enabled.

AWE API can be used even on the boxes with RAM below of a process’s VAS. In fact AWE could be used to avoid any type of paging (Remember that when locking pages in memory using VirtualLock mechanism Windows still can page out the whole process) When using AWE mechanism, the OS can’t interfere at all. With such flexibility comes a difficulty. Misuse of AWE mechanism can stall the whole box so that only way to recover from such state could be a reboot.

Well I think it is enough info for today J From this and previous posts all I want you is to grasp the concepts. They are really important! Here is the quick summary:

- Every process has its own VAS

- VAS is often neglected by developers.

- VAS is limited resource even on 64 bit platform

- VAS is managed by Windows the same way as one would manage a heap

- VAS's smallest region is 64kb

- VAS regions is managed by corresponding VAD in kernel

- VAD describes state of VAS region - allocated, committed, uncommitted

- A VAS page bound to physical page in RAM on demand, the first time page is accessed, unless running without page file).

- Physical pages can be private or shared

- Set of all private physical pages for a process is called private bytes (PM), or virtual memory size (TM)

- Set of private physical pages in RAM is called process working set.

- Physical pages bound to specific VAS region can be locked in memory

- PAE switch enables Windows’s support for 64GB of RAM

- On x86 VAS is limited to 2GB, 3GB or 4GB

- /3GB switch increases user VAS to 3GB and decreased kernel VAS to 1GB.

- /3GB switch limits amount of physical memory enabled through PAE switch down to 16GB

- /3GB switch is not recommended

- AWE is mechanism, it is not memory.

- AWE enables single process to use memory outside of VAS limits.

- /3GB swith is not required to enable AWE mechanism

- /PAE switch is not required to enabel AWE mechansim

Some interesting gotchas:

- Allocating physical memory using AWE mechanism is slow on Windows 2000. (This is a reason why SQL Server 2000 doesn’t manage memory dynamically when AWE is enabled). The problem is fixed in Windows 2003 server so that Yukon does allocates physical memory dynamically

- Physical pages allocated through AWE mechanism are not part of process working set neither large pages (I will cover large pages sometime latter). This is exact reason why neither PerfMon or Task Manager show them

Next time we will talk about memory pressures and then we will be ready to dive into SQL Server Memory Manager.

For now enjoy the weekend!

A look at Virtual Address Space - VAS

slavao — Thu, 27 Jan 2005 19:43:00 GMT

Memory is a set of common resources shared by applications, their components and operating system. One needs to be very specific when referring to a given memory resource. Mistakenly, in many cases, developers, users, DBAs refer to different memory resources using one generic term memory. This perfectly worked in DOS times when things were simple, when there was no virtual memory and when application didn’t have to share the computer box with others. Since then things have changed a lot.

There are several types of memory resources. The resources can be controlled by OS as well as by user code so it is very important for every developer to get full understanding of what those resources are.

As promised, today I would like to talk about Virtual Address Space, VAS. In my previous post, http://weblogs.asp.net/slavao/archive/2005/01/26/360759.aspx, I showed how one can track VAS in next release of SQL Server, Yukon. (Pretty cool aha?). VAS is a memory resource that most of time gets overlooked when dealing with memory issues. It often gets neglected by books when they describe memory manager. Even Windows's Task Manager doesn't have a VAS counter per process (bummer!). As it turns out understanding how Windows operates with VAS for a process is very crucial when dealing with memory issues.

Every process has its own VAS, simple right :)?. Windows creates separate VAS for a new process when processing CreateProcess API. It enables developers to allocate/free VAS regions by using VirtualAlloc*/VirtualFree* APIs. VAS region can be bound to physical memory right a way or latter on when needed. If you decide to commit VAS region right way, Operating System will first allocate VAS region and only then will go ahead and commit it. The minimal size of VAS region is 64kb (Remember this is very important!) Moreover all VAS regions are multiple of 64kb. In its turn the binding of VAS to physical memory is done base on underline page size, which could be 4kb or 8kb (There are also large pages 4MB and 16MB on x86 and IA64 but we will talk about them sometime latter). Here is very important point Any new allocation using VirtualAlloc* will be always rounded up to 64kb boundary so that if you allocate new VAS region bound to physical memory OS will consume amount of physical memory rounded up to page size and will consume VAS of the process rounded up to 64kb boundary ( When I learned it first time. This was first shock to me!)For example if you call VirtualAlloc for 1024 bytes with COMMIT flag. OS will allocate 64kb VAS's region in specified process and will allocate, commit, a page, 4kb or 8kb, of physical storage. The bottom line is that you need to be very careful when using VirtualAlloc* API. If you don’t use it you still need to fully understand its implications for debugging purposes for example it is more and more often on x86 platform when you are going to see process will be running low on or out of VAS. In order be able to troubleshoot such cases you have to know what VAS is and how it can be consumed. And guess what it is really possible to run out of VAS on 64bit :-). (In our test lab while working with large boxes we did run out of VAS multiple times and by the way I currently have a bug assigned to me with exact the same issue but on a smaller box, isn't this fun :-). The key here is when designing your application make sure that you don’t' scale your data structures with amount of physical RAM installed on the box)

As we said there is a VAS per process. Everything that gets allocated, loaded, created and mapped in your process uses VAS so that all of the above mechanism will use type of VirtualAlloc* call prior to whatever they are trying to do. For example loader before loading a dll into the process's VAS will call type of VirtualAlloc* to allocate VAS region for the dll; when creating thread inside of the process OS will first reserve VAS region for thread’s stack the same will happen when mapping file into the process space.

In order for component to use, allocate, VAS region they need to use some type of VirtualAlloc* call. Underneath Operating System allocates VAD, virtual address descriptor object, structure, that describes VAS region. VAD object fully describes VAS region, i.e. its size, allocation attributes and protection. As you might expect Operating System maintains VAD tree. VAD tree enables OS to efficiently provide VAS management. You can take a look at your process VAD tree using windbg's local kernel debugger. (Believe me it is cool!). Moreover, what do you think VirtualQuery API does? Yes you are right it walks process VAD tree and yes that is what we use in SQL Server to generate sys.dm_os_virtual_address_dump view. As you can see Windows has to manage a process's VAS the same way as you would manage any type of heap with smallest block of 64kb. (You can practice and write your own, but unfortunately there is no way to plug it in :-))

As we said VAS is very important resource and it is used by lots of different components loaded into your process space so that ability to track or account for it is very important. You can get idea of how much of VAS is currently consumed inside of your process by looking at perfmon’s virtual bytes counter. Task manager doesn’t have such counter.

Unfortunately Windows doesn’t provide simple way for tracking and accounting of VAS. OS not providing such mechanism doesn’t mean the mechanism can’t be built though. There are several tools that can be used to track VAS. One of them is vadump.exe. I believe VADUMP walks a process VAD tree utilizing type of VirtualQuery API. It provides you with information about VAS regions that you latter can analyze.

Even though VADUMP can give plenty of information in many cases it still is not enough to dig out VAS problem, i.e. leak. For many VAS regions VADUMP doesn’t provide developer with information to what component region belongs to. One way to get such information is to hook VirtualAlloc/VirtualFree (Do you think it is sufficient :)) call and then keep track of all VAS operations. This is exactly what LeakDiag tool does. It leverages detours library provided by Microsoft Research to hook VAS APIs. For every allocation it remembers allocation information along with stack. At any point in time the information could be dumped into file and analyzed. As far as I know LeakDiag is currently the only tool that can perform VAS tracking at such level, (I might be mistaken but it is always great to think that your tool is the only tool :-)). You can get LeakDiag from ftp://ftp.microsoft.com/PSS/Tools/Developer%20Support%20Tools/LeakDiag/

So here are some major points I really wanted to get across.

VAS is often neglected

VAS is limited resource even on 64bit platform

VAS is managed by Windows the same way as any heap

VAS's smallest region is 64kb

VAS leak causes process to exit (How can you leak VAS, Consider it to be your homework :-))

There are few tools to track VAS

I hope, these points make sense and you feel them now by heart :-). Understanding how VAS works is the first step in understanding SQL Server's memory manager, which is one of the most sophisticated servers out there, but you know what inside it is simple :-). Well this won't be a next topic of our discussion just yet. The next time I will probably continue talking about other types of memory resources and also different types of memory pressures (Pressures... of different types? Hmmm ... sounds interesting)

If you learn something from this post then I am really happy :-)!

Enjoy your day!