Slava Oks's WebLog : Q&A

Q & A: I can't make SQL Server 2005 32 bit version to use all memory on the machine

slavao — Thu, 03 Aug 2006 17:41:00 GMT

I’m currently using the RTM+SP1 x86-SQL2005 build on a server which has 32GB of physical memory. I notice that during ramp-up the memory usage (as seen under task manager) increases to about 28GB (i.e. “Available memory” reduces to 4GB) and stays there....(read more)

Q and A: Enabling AWE settings for SQL Server 32 bit edition in WOW

slavao — Wed, 12 Apr 2006 21:25:00 GMT

Q. A customer wants to run SQL 2000 SP4 in a Windows Server 2003 SP1 x64 edition. They did the installation without problems but SQL Server is not using more than 4GB of memory. Is it possible to enable AWE in this configuration? A. The answer is yes,...(read more)

Q and A: Some questions about output from dbcc memorystatus

slavao — Mon, 20 Mar 2006 04:18:00 GMT

Recently I have recieved following question: A customer of mine is getting this output on an Itanium with 16GB of memory: Memory Manager KB ------------------------------ -------------------- VM Reserved 16979888 VM Committed 217928 AWE Allocated 14116272...(read more)

Q and A: SQL Server memory allocated outside of Buffer Pool, formerly known as MemToLeave

slavao — Tue, 14 Mar 2006 21:14:00 GMT

Q. Under SQL Server 2000 Is the memtoleave concept still valid under 64-bit? A. Yes. If SQL Server’s component requests memory larger than 8 KB block, the block will be allocated outside of Buffer Pool. Remember, inside of SQL Server, Buffer Pool is preferable...(read more)

Q and A: Do I need to run "sp_configure 'awe enabled', 1" to make 64 bit version of SQL Server use locked pages?

slavao — Mon, 13 Mar 2006 20:44:00 GMT

The answer is No. The source of possible confusion came from one of my previous posts: http://blogs.msdn.com/slavao/archive/2005/11/15/493019.aspx . I apologize for the confusion. In order to make 64 bit version of SQL Server leverage locked pages all...(read more)

Q and A: Virtual Address Space, VAS, reservation - why bother?

slavao — Tue, 15 Nov 2005 23:09:00 GMT

Seems like today is a question day :-). I will attempt to answer several questions about VAS I got today:

Q1. There is one aspect of Virtual Memory that is just not clear yet. I think I understand about VAS, but maybe not as much as I could, but I don’t understand about ‘reserving memory’.

When we talk about SQL Server having a certain amount of reserved, but uncommitted, memory, are we talking about VAS?

A1: Yes, We are talking about Virtual Address Space, VAS. SQL Server also has a different, unrelated, notion of reservation. Whenever query execution engine needs buffers to run a complex query, it goes to Buffer Pool, BP, and reserves buffers from it. If there is not enough buffers in BP to serve the reservation, the query doesn't get run until either buffers become available or time out expires. The bottom line is that VAS reservation and BP reservation are two different things.

Q2. What is the value of reserving before committing? Why not just commit at the beginning, or wait until we need to commit in order to reserve?

A2. I understand these questions are related to VAS not to BP's reservations :-). There are several reasons why one would reserve VAS region first and commit it latter.

The first reason is that there are different ways of committing reserved VAS regions. In order to understand this, we need to understand the difference between reserving and committing. Reserving is a process of allocating VAS regions, committing is a process of binding of VAS's regions to a physical memory. However committing is not the only way to perform the binding. Once VAS region is reserved it can be bound to physical memory by means such as committing, AWE mechanism, file mapping API's and etc. In some cases, i.e. simple commit, OS allows to reserve VAS region and commit it in one shot using VirtualAlloc API. In other cases, i.e. AWE mapping, VAS regions have to be reserved upfront and only then they can be bound to physical memory allocated through AWE mechanism.

VAS's limitation is another reason for reserving VAS upfront and then committing its regions latter on demand. On x86 VAS is limited resource - its size can't exceed 4GB and in most of the case it is 2GB. Everyone requires VAS's regions in order use memory. The high demand makes VAS valuable resource. Consequently many components, including SQL Server's Buffer Pool, reserve VAS regions upfront and bind them to physical memory latter on demand. In most of the cases there is no reason to bind reserved VAS regions to physical memory right a way especially if only worry about VAS availability for latter use. Moreover in some cases the premature binding can introduce unnecessary physical memory pressure

Q3. Can the reserved memory and/or VAS grow or shrink the way committed memory can?

A3. Though reserved VAS regions can't grow, they can be freed when a component no longer needs them. For example SQL Server 2005 when configured to use AWE mechanism reserves VAS regions in 4MB regions. If SQL Server notices that it runs out free VAS regions - basically it hits VAS pressure, SQL Server will attempt to free unused 4MB regions to release the pressure.

Q4. What happens if SQL Server can’t get the physical memory that it has reserved?

A4. In cases when system low on physical memory - SQL Server might fail binding giving VAS region to the physical memory. In most of the cases it will result in some kind of OOM error. It the error happens in the engine ERROLOG will have information that SQL Server failed to commit a region of the given size. In addition there is OOM ring buffer that will have a record about the failure. Keep in mind that SQL Server has internal mechanism to detect the condition up front so in normal scenario it will attempt to release its physical memory usage before it hits the failure. In most of the cases such failure means that there is either interaction problem between SQL Server, OS, and/or some other application; someone's leaking physical memory or given configuration can't sustain applied load.

Hope this helps. Let me know if you have more questions.

Q and A: x64 & AWE mechanism

slavao — Tue, 15 Nov 2005 21:15:00 GMT

Today I received following question:

Hello,

We will be running Windows Server 2003 x64 with SQL Server 2005 x64. I would think AWE would not come into the picture in this scenario.

I'm trying to find out more about SQL Server 2005 memory management on x64 systems using x64 versions of the OS and SQL Server.

What role if any does AWE play in this scenario?

Thanks,

In the post http://blogs.msdn.com/slavao/archive/2005/08/31/458545.aspx I did indicate that on 64 bit platform we recommend to enable SQL Server to use locked pages in memory. You can do it by giving SQL Server's account a right to lock pages in memory. By giving this right to SQL Server you enable SQL Server to allocate and hold on to physical memory so that OS can't reclaim the memory back through paging - swapping SQL Server to disk. Even though we have plenty of safeguards inside of the server, still the swapping in some cases can bring SQL Server's performance to its knees.

When running with locked pages enabled, SQL Server monitors box wide memory state and does release memory back to OS in the case of pressure. The key here is that internally SQL Server makes its own decision about what memory can be freed. Internal decision making significantly helps SQL Server to avoid performance degradation in cases of box wide memory pressure. Keep in mind that SQL Server won't release its memory below 'min server memory' specified through sp_configure.

So where does AWE mechanism comes in here? On 64 bit platforms when running with lock pages privilege enabled, we use OS's AWE mechanism http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dngenlib/html/awewindata.asp to allocate physical memory. However we map allocated memory only once since we have plenty of VAS, virtual address space. On 64 bit platform to make use of locked pages you don't have to turn on AWE through sp_configure.

Hope this answers the question!

Q and A: Using Lock Pages In memory on 64 bit platform

slavao — Wed, 31 Aug 2005 19:59:00 GMT

Q: Hello Slava, I would like to confirm my understanding that on SQL 2005 64 bit edition it is recommended to grant Lock Pages in Memory right to the SQL account and then turn on the AWE setting. Thanks

A: Yes, we do recommend to turn on Lock pages in memory so that OS doesn't page SQL Server out. However on 64 bit you only need to grant the right "Lock Pages in Memory" to the SQL account for SQL Server to utilize this feature. You do need to to change any of AWE settings through sp_configure.

When you enable Lock Pages In memory, dbcc memorystatus's output will show that AWE mechanism is in use, Why? The reason for this is that for both 32 bit's AWE and 64 bit's Lock Pages we use Windows AWE mechanism described here http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dngenlib/html/awewindata.asp. The only difference is that on 64 bit we only have to map physical pages ones.

Q and A: Controlling size of a single cache in SQL 2005

slavao — Fri, 03 Jun 2005 17:08:00 GMT

Q: I have teaching a class on Yukon for the Ascend program, and the class had a question I wanted to double check before answering. On an Itanium with 1 TB of RAM, the class’ curiosity asked, “If plan cache is very large “Then how does Plan Cache lookups work such that the plan lookup does not itself become a bottleneck? The reason for the question is because the RM thread would not necessarily decrement the tick count on plans if they have not reached 50% of BPool (or am I wrong on that point?). So if there is 450 GB used for Plan Cache, what makes Yukon plan cache lookups work smarter than Shiloh's?

A: In my post about SQLOS caching http://blogs.msdn.com/slavao/archive/category/9005.aspx I have talked about clock algorithm and two clock hands - external and internal. I have mentioned that internal clock hand controls a size of the single cache " ...

You can think of internal clock hand as a way to put a max cap on a single cache. If this mechanism didn't exists then it would be possible for a single cache to push the whole process into memory pressure or become a bottleneck on lookups. For example if you execute numerous ad hoc queries, they can be cached. Without having internal clock hand or some kind of cap they would force the whole SQL Server's process into memory pressure or would cause severe performance degradations. To avoid such situation internal clock hand will starts moving once framework predicts that procedure cache's max cap is reached …" However in that post I forgot to mention that internal clock hand will start moving not only when specific size is reached but also when a given number of entries are put into the cache. The number of entries depends on max server memory setting, width of cache's hash table as well as memory state on the box. Keep in mind that this behavior affect all caches not only procedure cache - every cache has its own internal clock hand. If required every operation on a cache such as look up will move its internal clock hand.

One could observe movement of internal clock hands by querying sys.dm_os_memory_cache_clock_hands. You can also find out the width of the hash table by querying sys.dm_os_memory_cache_hash_tables

I hope your question is answered. I will be glad to answer more :-)

Q and A: Max memory with sql server 2000 / win2003 x64

slavao — Fri, 03 Jun 2005 16:06:00 GMT

Today, I received following question

Q: Slava,
I have a question for you. Sql 2000 is documented to support a maximum of 64GB when used with 2000 datacenter edition.
I assume this is the limit of 2000 Datacenter, but is it also a limit defined in sql server 2000?
We are purchasing a new server which can support a max of 128G of memory, and believe we have use for that much memory.
If we use an OS that can recognize it (such as 2003 enterprise x64), can sql server 2000 utilize 128G, or is it somehow limited to 64G?

A: I assume you are talking about SQL 2000 32 bit. Latest version of SQL 2000 SP4 won't be able to take advantage of memory more than 64GB. Moreover when running in WOW Windows supports max of 128GB on AMD64 and only 64GB on Intel x64. My general recommendation for installations running with large amount of memory is to run SQL Server natively on 64 bit platform. As you know, SQL 2000 only supports IA64, so you have couple of options:

- Get IA64 based hardware and run SQL2000 IA 64 version.

- Get x64, run SQL Server 2000 with 64GB in WOW. Once SQL 2005 is out, get it and run it natively.

One of the major reasons for not supporting 128GB is for buffer descriptors array becoming too big and we didn’t have a chance to test this configuration thoroughly. As you know buffer descriptors occupy process VAS and can’t be mapped / unmapped using AWE mechanisms. To support 128GB of amount of VAS consumed would have to be more than doubled. It means that just doubling VAS size, now its 4GB on x64's WOW, might not help.

Please let me know if you have more questions