Memory is a set of common resources shared by applications, their components and operating system. One needs to be very specific when referring to a given memory resource. Mistakenly, in many cases, developers, users, DBAs refer to different memory resources using one generic term memory. This perfectly worked in DOS times when things were simple, when there was no virtual memory and when application didn’t have to share the computer box with others. Since then things have changed a lot.
There are several types of memory resources. The resources can be controlled by OS as well as by user code so it is very important for every developer to get full understanding of what those resources are.
As promised, today I would like to talk about Virtual Address Space, VAS. In my previous post, http://weblogs.asp.net/slavao/archive/2005/01/26/360759.aspx, I showed how one can track VAS in next release of SQL Server, Yukon. (Pretty cool aha?). VAS is a memory resource that most of time gets overlooked when dealing with memory issues. It often gets neglected by books when they describe memory manager. Even Windows's Task Manager doesn't have a VAS counter per process (bummer!). As it turns out understanding how Windows operates with VAS for a process is very crucial when dealing with memory issues.
Every process has its own VAS, simple right :)?. Windows creates separate VAS for a new process when processing CreateProcess API. It enables developers to allocate/free VAS regions by using VirtualAlloc*/VirtualFree* APIs. VAS region can be bound to physical memory right a way or latter on when needed. If you decide to commit VAS region right way, Operating System will first allocate VAS region and only then will go ahead and commit it. The minimal size of VAS region is 64kb (Remember this is very important!) Moreover all VAS regions are multiple of 64kb. In its turn the binding of VAS to physical memory is done base on underline page size, which could be 4kb or 8kb (There are also large pages 4MB and 16MB on x86 and IA64 but we will talk about them sometime latter). Here is very important point Any new allocation using VirtualAlloc* will be always rounded up to 64kb boundary so that if you allocate new VAS region bound to physical memory OS will consume amount of physical memory rounded up to page size and will consume VAS of the process rounded up to 64kb boundary ( When I learned it first time. This was first shock to me!)For example if you call VirtualAlloc for 1024 bytes with COMMIT flag. OS will allocate 64kb VAS's region in specified process and will allocate, commit, a page, 4kb or 8kb, of physical storage. The bottom line is that you need to be very careful when using VirtualAlloc* API. If you don’t use it you still need to fully understand its implications for debugging purposes for example it is more and more often on x86 platform when you are going to see process will be running low on or out of VAS. In order be able to troubleshoot such cases you have to know what VAS is and how it can be consumed. And guess what it is really possible to run out of VAS on 64bit :-). (In our test lab while working with large boxes we did run out of VAS multiple times and by the way I currently have a bug assigned to me with exact the same issue but on a smaller box, isn't this fun :-). The key here is when designing your application make sure that you don’t' scale your data structures with amount of physical RAM installed on the box)
As we said there is a VAS per process. Everything that gets allocated, loaded, created and mapped in your process uses VAS so that all of the above mechanism will use type of VirtualAlloc* call prior to whatever they are trying to do. For example loader before loading a dll into the process's VAS will call type of VirtualAlloc* to allocate VAS region for the dll; when creating thread inside of the process OS will first reserve VAS region for thread’s stack the same will happen when mapping file into the process space.
In order for component to use, allocate, VAS region they need to use some type of VirtualAlloc* call. Underneath Operating System allocates VAD, virtual address descriptor object, structure, that describes VAS region. VAD object fully describes VAS region, i.e. its size, allocation attributes and protection. As you might expect Operating System maintains VAD tree. VAD tree enables OS to efficiently provide VAS management. You can take a look at your process VAD tree using windbg's local kernel debugger. (Believe me it is cool!). Moreover, what do you think VirtualQuery API does? Yes you are right it walks process VAD tree and yes that is what we use in SQL Server to generate sys.dm_os_virtual_address_dump view. As you can see Windows has to manage a process's VAS the same way as you would manage any type of heap with smallest block of 64kb. (You can practice and write your own, but unfortunately there is no way to plug it in :-))
As we said VAS is very important resource and it is used by lots of different components loaded into your process space so that ability to track or account for it is very important. You can get idea of how much of VAS is currently consumed inside of your process by looking at perfmon’s virtual bytes counter. Task manager doesn’t have such counter.
Unfortunately Windows doesn’t provide simple way for tracking and accounting of VAS. OS not providing such mechanism doesn’t mean the mechanism can’t be built though. There are several tools that can be used to track VAS. One of them is vadump.exe. I believe VADUMP walks a process VAD tree utilizing type of VirtualQuery API. It provides you with information about VAS regions that you latter can analyze.
Even though VADUMP can give plenty of information in many cases it still is not enough to dig out VAS problem, i.e. leak. For many VAS regions VADUMP doesn’t provide developer with information to what component region belongs to. One way to get such information is to hook VirtualAlloc/VirtualFree (Do you think it is sufficient :)) call and then keep track of all VAS operations. This is exactly what LeakDiag tool does. It leverages detours library provided by Microsoft Research to hook VAS APIs. For every allocation it remembers allocation information along with stack. At any point in time the information could be dumped into file and analyzed. As far as I know LeakDiag is currently the only tool that can perform VAS tracking at such level, (I might be mistaken but it is always great to think that your tool is the only tool :-)). You can get LeakDiag from ftp://ftp.microsoft.com/PSS/Tools/Developer%20Support%20Tools/LeakDiag/
So here are some major points I really wanted to get across.
VAS is often neglected
VAS is limited resource even on 64bit platform
VAS is managed by Windows the same way as any heap
VAS's smallest region is 64kb
VAS leak causes process to exit (How can you leak VAS, Consider it to be your homework :-))
There are few tools to track VAS
I hope, these points make sense and you feel them now by heart :-). Understanding how VAS works is the first step in understanding SQL Server's memory manager, which is one of the most sophisticated servers out there, but you know what inside it is simple :-). Well this won't be a next topic of our discussion just yet. The next time I will probably continue talking about other types of memory resources and also different types of memory pressures (Pressures... of different types? Hmmm ... sounds interesting)
If you learn something from this post then I am really happy :-)!
Enjoy your day!