Slava Oks's WebLog : SQL Server Scheduling

SQLOSDMV's Continue

slavao — Tue, 14 Nov 2006 21:26:00 GMT

sys.dm_os_waiting_tasks One can run lots of interesting queries using this view. You can even use this view to perform deadlock detection that is not resolvable by deadlock monitor, DM. For example if you have tasks waiting on external resources such...(read more)

SQLOS's DMVs Continue

slavao — Fri, 29 Sep 2006 05:22:00 GMT

sys.dm_os_schedulers Q. Do I need to by more CPUs? In order to answer this question you have to find out if your load is really CPU bounded. Your load is really CPU bounded if a number of runnable tasks per each scheduler always greater than 1 and all...(read more)

Talking points around SQL Server 2005 scalability

slavao — Thu, 03 Aug 2006 18:09:00 GMT

Couple of weeks ago I did a talk on SQL Server 2005 scalability. The actual talk was not about the enormous amount of features that you can leverage in SQL Serve 2005 to make your application scale but rather how SQL 2005 achieves its scalability. In...(read more)

Set of new features in SQL Server 2005 enabled by SQLOS

slavao — Wed, 12 Apr 2006 21:57:00 GMT

Couple of weeks ago I had a chance to visit set of our customers and talk to them directly. It was an unforgettable experience. As a part of the visit I made set of presentations related to SQLOS and new features it enables in SQL Server 2005. I thought...(read more)

Be aware: To Hyper or not to Hyper

slavao — Sat, 12 Nov 2005 20:48:00 GMT

Our customers observed very interesting behavior on high end Hyperthreading (HT) enabled hardware. They noticed that in some cases when high load is applied SQL Server CPU usage increases significantly but SQL Server performance degrades. Occasionally they would also see message in errorlog indicating one of the threads can't acquire a spinlock (Spinlock is a lightweight synchronization data structure. Threads keep a spinlock for short period of time. If a thread can't acquire a spinlock it will spin in a tight loop waiting for spinlock to become available). Customers also noticed that when HT is disabled under the same circumstances CPU usage increases slightly and performance is at appropriate level. What is going on here?

After some pondering and investigation we came up with the following theory. Database server has different type of threads: worker threads, threads executing clients request, and system threads that performing system tasks. An example of a system thread is a lazywriter. The responsibility of the lazywriter is to lazily scan through database buffers and push least recently used buffers out of the cache. Lazywriter usually kicks in whenever data page cache is full and server short of free data page buffers. The interesting behavior of lazywriter is that it scans large amount of memory. With Intel HT technogolgy logical processors share L1 & L2 caches. As you would guess lazywriter's behavior can potentially trash L1 & L2 caches. If a worker thread end up running on logical CPU that shares physical CPU with lazywriter its cache will be constantly trashed. It means that most of the memory accesses for the worker thread will be L1 & L2 misses. Moreover whenever the worker attempts to access hot data structure protected by spinlock it potentially might have a less chance of acquiring the spinlock which will cause the worker's spinning. This behavior will translate to high CPU utilization and significant drop in performance.

To confirm the theory I decided to create an experiment. I wrote a test program which consist of two types of threads: scan threads and lock threads. Scan threads are scanning through large amounts of memory. Lock threads constantly attempt to acquire global lock. If the theory is correct a lock thread that shares a physical CPU with a scan thread should acquire lock much less often than other threads in a given period of time. Below is the program's listing. (Disclosure: The program is for illustrative purpose only, it was written to illustrate the observed behavior, it shouldn't be treated as a final-well tested product :-))

#include <stdio.h>

#include <stdlib.h>

#include <windows.h>

#include <eh.h>

#include <wtypes.h>

#include <locale.h>

#include <process.h>

extern "C" void _mm_pause(void);

#pragma intrinsic (_mm_pause)

inline void

SmtPause ()

{

#if defined (_X86_) || defined (_AMD64_)

// PAUSE instruction (REP NOP) helps SMT scheduling

_mm_pause ();

#endif // _X86_

}

// Statistic information for threads

struct Statistics

{

int locked;

int spinned;

DWORD_PTR affinity;

};

// Type of threads the tests using

enum ThreadTypes

{

SCAN_THREAD,

LOCK_THREAD,

LAST_THREAD

};

// Entry Point

typedef void( __cdecl *ThreadEntryPoint )( void * );

// Inforamtion per thread - enables us to create threads in a common way

struct ThreadInfo

{

ThreadEntryPoint func;

LONG_PTR affinity;

};

// Global constants

// Number of CPUs

const int MaxNumberOfCPUs = 64;

// Block of memory to scan by scan thread

char MemoryBlock [1024 *1024];

// Stat information: number of threads is limmited by number of CPU,

// that is the purpose of the test

volatile Statistics Stats [MaxNumberOfCPUs];

// Number of threads running

volatile long NumberOfThreadsRunning = 0;

// "Spinlock" that all lock threads attempt to acquire

volatile long Lock = 0;

//-----------------------------------------------------------------------------

// Function: scanThread

// Description:

// Attempt to invalidate caches

// Notes:

// 1. Function changes 1MB of contiguous vm memory which appear

// to be enought invalidate L1&L2 caches

void __cdecl

scanThread (void* param)

{

char i = 0;

ULONG_PTR threadAffinity =(DWORD_PTR) param;

long threadId = InterlockedIncrement (&NumberOfThreadsRunning) -1;

SetThreadAffinityMask (GetCurrentThread (),threadAffinity);

// Initialize affinity

Stats [threadId].affinity = threadAffinity;

while (i >= 0)

{

// Memset memory: quick and dirty scan

memset (MemoryBlock, 0, sizeof (MemoryBlock));

Stats [threadId].spinned++;

}

return;

}

//-----------------------------------------------------------------------------

// Function: lockThread

// Description:

// Acquires and releases a lock in a loop

// Notes:

void __cdecl

lockThread (void* param)

{

ULONG_PTR threadAffinity =(DWORD_PTR) param;

long threadId = InterlockedIncrement (&NumberOfThreadsRunning)-1;

SetThreadAffinityMask (GetCurrentThread (), threadAffinity);

// Initialize affinity

Stats [threadId].affinity = threadAffinity;

while (1)

{

if (Lock == 0 && InterlockedExchange (&Lock, 1) == 0)

{

// Increment number of times we acquired lock

Stats [threadId].locked ++;

Lock = 0;

}

else

{

Stats [threadId].spinned++;

// Be HT friendly

SmtPause ();

continue;

}

return;

}

//-----------------------------------------------------------------------------

// Function:main

// Description:

// Creates lock and scan threads, reporting stats every 10sec

// Notes:

int __cdecl

main (

int argc,

char* argv [])

{

LONG_PTR threadId = 0;

ThreadInfo threadInfo [LAST_THREAD];

DWORD_PTR processAffinity;

DWORD_PTR threadAffinity;

DWORD_PTR systemAffinity;

int counter = 0;

int numberOfThreads = 0;

// Retrive process affinity

if (GetProcessAffinityMask (GetCurrentProcess (),

&processAffinity,

&systemAffinity) == FALSE)

{

printf ("GetProcessAffinityMask failed - exiting\n");

exit (-1);

}

// Initialize threads Information: First default and then check

// if client passed anything in

// Allocate all but one CPUs for threads running locking code

threadInfo [LOCK_THREAD].func = lockThread;

threadInfo [LOCK_THREAD].affinity = processAffinity & processAffinity-1;

// Allocate one CPU for scan thread

threadInfo [SCAN_THREAD].func = scanThread;

threadInfo [SCAN_THREAD].affinity =threadInfo [LOCK_THREAD].affinity ^ processAffinity;

// Read command line arguments: in simple way

for (counter = 0 ; counter< argc; counter++)

{

switch (*(argv [counter] + 1))

{

case 's':

case 'S':

{

// Retrieve affinity mask for scan threads

threadInfo [SCAN_THREAD].affinity =

(DWORD_PTR) _atoi64 ((argv [counter] + 2));

break;

}

case 'l':

case 'L':

{

// Retrieve affinity mask for lock threads

threadInfo [LOCK_THREAD].affinity =

(DWORD_PTR) _atoi64 ((argv [counter] + 2));

break;

}

case '?':

{

printf ("Usage: test -sn -lm \n"

"Example: test.exe -s8 -l7 (runs 1 scanning thread and 3 locking threads)\n"

"-sn is affinity of threads performing memory scan \n"

"-sm is affinity of threads acquiring lock");

exit (0);

}

default:

{

break;

}

// Start threads of all types based on supplied affinities

for (counter = 0; counter < LAST_THREAD; counter++)

{

while (threadInfo [counter].affinity != 0)

{

// Separate affiinity for a thread we are trying to

// create from all affinity mask

// Remember full mask

threadAffinity = threadInfo [counter].affinity;

// Turn off least significant bit

threadInfo [counter].affinity &= threadInfo [counter].affinity -1;

// Calculate the affinity bit for the thread:

// Leave one bit on - affinity bit

threadAffinity ^= threadInfo [counter].affinity;

// Use beginthread for simplicity

if (_beginthread (threadInfo [counter].func,

NULL,

(void*) threadAffinity) == -1)

{

printf ("Failed to create thread - exiting\n");

exit (-1);

}

numberOfThreads++;

}

// Loop and report stats periodically

while (1)

{

// Report every 10 sec

Sleep (10000);

for (threadId=0; threadId < numberOfThreads; threadId++)

{

printf ("Thread Id = %d\t, Thread Affinity = %p\t, Locked = %d\t, Spin = %d\t\n ",

threadId,

Stats [threadId].affinity,

Stats [threadId].locked,

Stats [threadId].spinned);

// Reset run time data

Stats [threadId].locked = 0;

Stats [threadId].spinned =0;

}

return 0;

}

As you can see one can specify an affinity for scan and lock threads at start up of the program.

I ran sever experiments on my dev box which has 2 physical and 4 logical CPUs. CPU 1 (Affinity 0x1) & 3 (Affinity 0x4) and 2 (Affinity 0x2) & 4 (Affinity 0x8) share physical CPUs respectively. The box also has Windows 2003 Server SP1 on it. Below are the results:

test.exe -s8 -l7

Thread Id = 0 , Thread Affinity = 00000008 , Locked = 0 , Spin = 9035

Thread Id = 1 , Thread Affinity = 00000001 , Locked = 9290331 , Spin = 10373717

Thread Id = 2 , Thread Affinity = 00000002 , Locked = 676397 , Spin = 40879794

Thread Id = 3 , Thread Affinity = 00000004 , Locked = 10274030 , Spin = 42433521

test.exe -s4 -l11

Thread Id = 0 , Thread Affinity = 00000004 , Locked = 0 , Spin = 8943

Thread Id = 1 , Thread Affinity = 00000001 , Locked = 741166 , Spin = 11421139

Thread Id = 2 , Thread Affinity = 00000002 , Locked = 10747824 , Spin = 10024252

Thread Id = 3 , Thread Affinity = 00000008 , Locked = 10210221 , Spin = 13988134

test.exe -s1 -l14

Thread Id = 0 , Thread Affinity = 00000001 , Locked = 0 , Spin = 8472

Thread Id = 1 , Thread Affinity = 00000002 , Locked = 10550011 , Spin = 20156206

Thread Id = 2 , Thread Affinity = 00000004 , Locked = 722554 , Spin = 11211074

Thread Id = 3 , Thread Affinity = 00000008 , Locked = 11182506 , Spin = 25376166

test.exe -s2 -l13

Thread Id = 0 , Thread Affinity = 00000002 , Locked = 0 , Spin = 8585

Thread Id = 1 , Thread Affinity = 00000001 , Locked = 9900899 , Spin = 12267885

Thread Id = 2 , Thread Affinity = 00000004 , Locked = 8984095 , Spin = 11909080

Thread Id = 3 , Thread Affinity = 00000008 , Locked = 1297138 , Spin = 28199769

Great! The experiment confirms the theory. So does it mean you have to disable HT when using SQL Server? The answer is it really depends on the load and hardware you are using.

You have to test your application with HT on and off under heavy loads to understand HT's implications.

Keep in mind that not only lazywriter thread can cause slowdown but any thread that performs large memory scan - for example a worker thread that scans large amount of data might be a culprit as well.

For some customer applications when disabling HT we saw 10% increase in performance. So make sure that you do your home work before you decide to hyper on not to hyper :-)

Hope this information was useful. Let me know if you have any questions!

Configuring SQL Server 2005 for Soft NUMA

slavao — Fri, 19 Aug 2005 02:30:00 GMT

Sometimes I am surprised how issues come up the moment you mention them J. Recently, I have discussed Soft NUMA configurations in the article http://blogs.msdn.com/slavao/articles/441058.aspx. This week one of our customers had an interesting problem. The customer wanted to partition single SQL server instance based on the load. Customer’s application is heterogynous. It consists of TPCH type queries and data loading applications. The customer has a system, which is NUMA, with 2 nodes and 4 CPU per Node. The customer wanted to give the loading application two CPUs and the rest of CPUs to the queries. Is it possible to achieve it?

As you might guess the answer is SQL2005's Soft NUMA support. We advised them to configure SQL Server and clients to treat system as three node NUMA system. (Surprised? Yes it is possible with SQL 2005 Soft NUMA support). The configuration looks like following: zero node has 4 CPUs, first node has 2 CPUs and last node has 2 CPUs. Keep in mind, when you configure SQL Server for Soft NUMA, soft nodes should fully be contained in the real nodes, i.e. a soft node can not span several real NUMA nodes. Customer’s TPCH queries were configured to utilize zero and first nodes and the load application was configured to utilize last node. Once configured and started this configuration worked as expected – load was fully partitioned across CPUs. It is important to notice that the customer was very delighted with the experience. Below is the example of node & network configuration we provided customer with:

A. Node configuration – provides node and their affinity information.

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\90\NodeConfiguration\Node0]

"CPUMask"=dword:0000000F

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\90\NodeConfiguration\Node1]

"CPUMask"=dword:00000030

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\90\NodeConfiguration\Node2]

"CPUMask"=dword:000000C0

- Keep in mind affinity mask for a soft node can’t span multiple hardware nodes. For example 000000CF is incorrect mask.

B. Network Configuration – provides port & node assignment.

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL.1\MSSQLServer\SuperSocketNetLib\Tcp\IPAll]

"TcpPort"="1433[0x3],2000[0x4]"

"TcpDynamicPorts"=""

"DisplayName"="Any IP Address"

- This configuration makes SQL Server to listen on two ports. One port serves two nodes. It means that port node relations ship could be one two many.

- Keep in mind that in the square brackets you specify node affinity not the CPU affinity, i.e. 3 in this case means that port 1433 will processes requests on node 0 and 1

- When port is assigned to multiple nodes it will assign connections to the nodes in a round robin fashion

Once registry is configured server have to be restarted for changes to take affect. In addition clients should be configured to connect to specific ports. In the case of our customer TPCH clients were directed to port 1433 and the loading application was assigned to port 2000.

SQL Server 2005 supports more complex configuration for Soft NUMA. Below is information along with an example provided to me by our performance team; describing how to configure SQL Server 2005 for Soft NUMA in general. Please notice that you can actually bind a specific NIC to the node.

This is default key where you need to change the value of TcpPort.

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL.1\MSSQLServer\SuperSocketNetLib\Tcp\IPAll]

"TcpPort"=""

"TcpDynamicPorts"=""

"DisplayName"="Any IP Address"

Thus if you need 3 ports (2000,2001,1433) on a four node machine such that 2000 controls connections such that they go nodes 0 and 1 ,2001 causes them to go to nodes 2 and 3 and 1433 to go to all nodes You use:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL.1\MSSQLServer\SuperSocketNetLib\Tcp\IPAll]

"TcpPort"="2000[0x3],2001[0xc],1433"

Thus the values in the square brackets are node masks and NOT processor masks. The above key makes all NICs on the system listen to all the above ports. Now instead, if we want to control things on a NIC by NIC basis, we need to flip this switch

From

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL.1\MSSQLServer\SuperSocketNetLib\Tcp]

"ListenOnAllIPs"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL.1\MSSQLServer\SuperSocketNetLib\Tcp]

"ListenOnAllIPs"=dword:00000000

And then if we had Two NICs on the system besides the loopback adaptor. Note the Last part of the key “IP3” depend on the NIC number

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL.1\MSSQLServer\SuperSocketNetLib\Tcp\IP2]

"Enabled"=dword:00000001

"Active"=dword:00000001

"TcpPort"="2000[0x3],1433"

"TcpDynamicPorts"=""

"DisplayName"="Specific IP Address"

"IpAddress"="10.192.168.01"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL.1\MSSQLServer\SuperSocketNetLib\Tcp\IP3]

"Enabled"=dword:00000001

"Active"=dword:00000001

"TcpPort"="2001[0xc],1433"

"TcpDynamicPorts"=""

"DisplayName"="Specific IP Address"

"IpAddress"="10.192.168.02"

This means we shall listen to 2000 only 10.192.168.01 and pass connections received from 2000 only to nodes 0 and 1. And similarly for 2001.

Below configuration which makes an 8 proc 2 node system behave as an 8 node system. It assumes that there are 8 NICs in the system

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL.1\MSSQLServer\SuperSocketNetLib\Tcp]

"Enabled"=dword:00000001

"ListenOnAllIPs"=dword:00000000

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL.1\MSSQLServer\SuperSocketNetLib\Tcp\IPAll]

"TcpPort"=""

"TcpDynamicPorts"="2001[0x1],2002[0x2],2003[0x4],2004[0x8],2005[0x10],2006[0x20],2007[0x40],2008[0x80]"

"DisplayName"="Any IP Address"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL.1\MSSQLServer\SuperSocketNetLib\Tcp\IP3]

"Enabled"=dword:00000001

"Active"=dword:00000001

"TcpPort"="2001[0x1]"

"TcpDynamicPorts"=""

"DisplayName"="Specific IP Address"

"IpAddress"="10.192.168.01"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL.1\MSSQLServer\SuperSocketNetLib\Tcp\IP4]

"Enabled"=dword:00000001

"Active"=dword:00000001

"TcpPort"="2002[0x2]"

"TcpDynamicPorts"=""

"DisplayName"="Specific IP Address"

"IpAddress"="10.192.168.02"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL.1\MSSQLServer\SuperSocketNetLib\Tcp\IP5]

"Enabled"=dword:00000001

"Active"=dword:00000001

"TcpPort"="2003[0x4]"

"TcpDynamicPorts"=""

"DisplayName"="Specific IP Address"

"IpAddress"="10.192.168.03"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL.1\MSSQLServer\SuperSocketNetLib\Tcp\IP6]

"Enabled"=dword:00000001

"Active"=dword:00000001

"TcpPort"="2004[0x8]"

"TcpDynamicPorts"=""

"DisplayName"="Specific IP Address"

"IpAddress"="10.192.168.04"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL.1\MSSQLServer\SuperSocketNetLib\Tcp\IP7]

"Enabled"=dword:00000001

"Active"=dword:00000001

"TcpPort"="2005[0x10]"

"TcpDynamicPorts"=""

"DisplayName"="Specific IP Address"

"IpAddress"="10.192.168.05"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL.1\MSSQLServer\SuperSocketNetLib\Tcp\IP8]

"Enabled"=dword:00000001

"Active"=dword:00000001

"TcpPort"="2006[0x20]"

"TcpDynamicPorts"=""

"DisplayName"="Specific IP Address"

"IpAddress"="10.192.168.06"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL.1\MSSQLServer\SuperSocketNetLib\Tcp\IP9]

"Enabled"=dword:00000001

"Active"=dword:00000001

"TcpPort"="2007[0x40]"

"TcpDynamicPorts"=""

"DisplayName"="Specific IP Address"

"IpAddress"="10.192.168.07"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL.1\MSSQLServer\SuperSocketNetLib\Tcp\IP10]

"Enabled"=dword:00000001

"Active"=dword:00000001

"TcpPort"="2008[0x80]"

"TcpDynamicPorts"=""

"DisplayName"="Specific IP Address"

"IpAddress"="10.192.168.08"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\90\NodeConfiguration]

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\90\NodeConfiguration\Node0]

"CPUMask"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\90\NodeConfiguration\Node1]

"CPUMask"=dword:00000002

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\90\NodeConfiguration\Node2]

"CPUMask"=dword:00000004

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\90\NodeConfiguration\Node3]

"CPUMask"=dword:00000008

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\90\NodeConfiguration\Node4]

"CPUMask"=dword:00000010

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\90\NodeConfiguration\Node5]

"CPUMask"=dword:00000020

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\90\NodeConfiguration\Node6]

"CPUMask"=dword:00000040

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\90\NodeConfiguration\Node7]

"CPUMask"=dword:00000080

SQL Server 2005 NUMA support & troubleshooting

slavao — Tue, 02 Aug 2005 18:43:00 GMT

SQL Server 2005 has been designed to take advantage of NUMA features exposed by both hardware and the Windows operating system. There are several caveats that you need to be aware when attempting to run SQL Server on NUMA platforms.

In this post I would like to go over both Windows's and SQL Server's NUMA support, describe their possible configuration, and give some advises on troubleshooting some of the issues.

The latest SQL Server's CTP release has most of the NUMA support already built in so that you can play with it and see for yourself actual NUMA in action. If you are more interested in troubleshooting without understanding the actual issues you could jump to troubleshooting section right a way.

Background: I expect you understand classic cc-NUMA configuration so that I don't have to go into details of explaining them.

Windows NUMA Support

NUMA hardware platforms could be configured differently. The two major configurations are pure NUMA or Interleaved NUMA. Pure NUMA will appear to operating system as a set of CPU nodes, sometimes called pods, with local memory. Depending on a hardware manufacture there will be a different penalty to access memory residing on other nodes - remote memory. In this configuration for an application to perform well it needs to minimize a number of remote accesses.

When system is configured to use interleaved memory, to operating system it appears as a large SMP box where memory for cache lines is interleaved from different NUMA's nodes. This configuration is suitable for applications that don't have NUMA optimizations and might behave badly in pure NUMA configuration. For example if you don't have SQL2000 post SP3 QFE that has NUMA optimizations you shouldn't be running SQL Server on the pure NUMA configuration. In general SQL2000 doesn't have many NUMA optimization so that you might want always to configure system as interleaved.

When Windows starts up on pure NUMA hardware it recognizes system multi node configuration and boots accordingly. From our experience we have noticed that during the start OS will allocate memory mostly from a single node - keep in mind that this does depend on memory availability on the nodes as well as a number of applications OS has to start. This allocation pattern during OS startup could be problematic for a NUMA aware application - since memory is not distributed evenly across the nodes and there is no way to find out memory distributions across the nodes. As a system continue running the problem becomes more severe - there is less and less free memory available on the nodes. It could be due to memory hungry applications or to System File Cache, SFC. Depending on the system configuration it is possible for System File Cache to allocate significant amount of memory. Moreover on NUMA configurations it is quite possible for SFC to unevenly use memory across nodes. If a NUMA aware application only attempts to allocate local memory it might become a victim of memory starvation on the nodes in which it collided with SFC or with other memory hungry applications.

Windows exposes a set of APIs that allows applications to take advantage of NUMA:

GetNumaHighestNodeNumber

GetNumaProcessorNode

GetNumaNodeProcessorMask

GetNumaAvailableMemoryNode

There are several API's "gotchas" that you have to be aware of:

There is no explicit way to allocate memory from the given node - a- thread allocates a memory from a given memory node by changing its affinity to the given node and then calling VirtualAlloc (if running with swap file you will have to touch virtual address before it gets bound to physical page) or other low level APIs that will trigger physical page allocation. If OS doesn't have memory on a given node it will serve the memory from the different node. It won't fail the allocation.
GetNumaAvailableMemoryNode could return 0 even though there could be a still memory that OS could serve from the node.
Windows will serve memory in the following order: 1. From the node a thread is affinitized to 2. From node that has memory free and 3. From any node - memory freed up from standby lists, working sets and System File Cache.

The consequences of A & C is that NUMA aware applications should have more of special handling for remote memory. A & C might cause OS to return memory in random order such as remote, local, remote, and etc. Keep in mind if applications decides not to cache remote memory it might get in trouble because when memory is freed it gets put on the free list and will be given away on the next allocation request.

The consequences of B is that applications can't reliably decide whether they can or can't allocate memory from the node. Case B usually occurs after system runs for some time especially when large applications are present.

SQL Server 2005 NUMA Support

SQL Server 2005 attempts to take full advantage of NUMA architecture by leveraging SQLOS for more info see http://blogs.msdn.com/slavao/articles/441058.aspx. During startup SQL Server will configure itself base on underlying OS and hardware configuration. It will create the environment inside of itself to mimic the actual hardware - SQL Server will create a software abstraction, for purpose of our discussion we will call it a Node, around every NUMA node and its memory.

Every SQL Server's Node has a memory manager, set of schedulers each of which maps to a underlying CPU; I/O port and other components. SQL Server's Node could be considered as a separate SQL Server instance with its own I/O port that it listens on. Clients or separate applications can be configured to connect to a specific Node only. A Node can be online - accepting new work or offline, accepting neither new work or connections. State of a Node could be changed on the fly using affinity settings - see setting affinity below. A connection could be made to a given Node or set of Nodes - a network port could be associated with multiple Nodes. If no extra configuration specified a new connection is assigned to a next Node in a round robin fashion. If connection is made to a set of Nodes it will be assigned to a next Node in a round robin fashion across that set of Nodes. BOL has plenty of information on how to configure SQL Server's ports, nodes and clients to connect. Once connection is created it is bound to a Node until disconnected. If connection's Node is moved offline its work will be scheduled on other Nodes. All the memory that is required to serve the connection's needs is allocated locally from the memory that is attached to the Node. Database pages are always allocated locally from the Node's memory they accessed on unless they have been brought into the memory by connection assigned to another Node. Please keep in mind that in the latest CTP SQL Server attempts not to allocate memory from different Node. This behavior might cause premature OOM or significant query slow down.

To support different NUMA features SQL Server can be configured both on server and client site. On the server site DBAs can configure amount of memory, active Nodes, a number of threads, locked pages, network configuration - what NIC is bound to what Node. On the client side DBA can configure particular clients/applications to connect to specific Nodes.

Below are two settings that affect SQL Server behavior on NUMA

Affinity - Using sp_configure a DBA can easily change SQL Server affinity. The major difference between SQL Server 2000 and SQL Server 2005 is that affinity changes will take affect right away - no server restart is required. As you know from SQL 2000 if affinity is not set SQL Server will affinitize itself to all CPUs. In SQL Server 2005 when running on NUMA and affinity is not set SQL Server will affinitize each scheduler to a set of CPUs that a SQL Server Node maps to. For example for a NUMA node 1 (nodes are numbered starting with 0) with 4 CPUs, SQL Server's schedulers belonging to Node 1 will have affinity of 0xF0. When DBA does specify affinity, we call it hard affinity, every scheduler in a Node will be affinitized to its own CPU. For example in the above example first scheduler will have affinity 0x10. When a Node's scheduler maps to an underlying CPU we say that scheduler is online. It is possible for a scheduler to be offline. For example lets say on two NUMA node system, DBA sets server affinity to be 0x1F - only 5 schedulers a mapped to actual CPUs; 4 from the Node 0 and 1 from the Node 1. Last 3 schedulers from the Node 1 are moved offline - no new work will be submitted to them. If all schedulers from a given Node are offline then the whole Node is offline - it won't get any of work assigned moreover no new connections will be made to that Node.

Max Server Memory - These are the global settings. In the current CTP SQL Server 2005 will continue allocate memory from a given node until either node still has memory - allocation returns a memory from requested node or overall allocated memory doesn't exceed max server memory. SQL Server won't try to allocate/use remote memory - the issue will be addressed in the final version of SQL Server 2005.

Troubleshooting problems related to SQL Server running on NUMA

A query runs sporadically slow even if plan doesn't change or clients observes other significant perf degradations - is the most common problem. The major cause of the problem is memory exhaustion on the Node query is running on. The memory exhaustion could happen due to multiple reasons:

Node 0 problem - All memory on this node is consumed during OS startup.

Possible solution for this problem is to change SQL Server affinity to move Node 0 offline.

Node n problem - All memory on this node is consumed by other applications or by System File Cache (also see misconfigured SFC problem below). You can verify if you are hitting this problem by looking at the size of SFC, other applications' memory consumption and amount of memory consumed by SQL Server on every Node from dbcc memorystatus.

Possible solution for this problem is to grab all the memory for SQL Server before starting other applications. You can achieve it by utilizing max server memory, dynamic scheduling and clients affinity. The idea is to start one Node at a time, connect to the node, apply load to allocate memory on the Node, start next Node, apply load, force memory allocation and so on. At the end set min and max memory to be the same so that SQL Server doesn't shrink its memory usage under the load. Here are the steps you need to make to achieve it:

Configure SQL Server to run on multiple Nodes with a port per Node
Configure a client for each Node
Prepare large query, for example reindex, that will require at least DesirableMemory/#Nodes of memory
Start SQL Server
Move all Nodes offline except for Node 0 using sp_configure
Set "max server memory" to DesirableMemory/#Nodes using sp_configure
Connect to Node 0
Run the job
Increase "max server memory to "max server memory" + DesirableMaxMemory/#Nodes using sp_configure
Move next Node online using sp_configure
Repeat 7-10 for the rest of the Nodes
Set "max server memory" = "min server memory" = DesirableMaxMemory

Misconfigured System File Cache problem - This problem is related to Node n problem. If FSC is incorrectly configured it can consume all the memory on the machine. You can verify if you are hitting this issue by looking at the size of FSC through either TaskManager or perfmon. You can fix the issue by going to "My Computer"->"Properties"->"Advanced"->"Performance Settings"-> "Advanced"->"Memory Usage Programs"->"Ok" and reboot the box, if you changed the setting. It is possible for the applications even when configured for Programs to force large consumption in System File Cache. If it happens you could write an application that periodically trims the size of System File Cache. There is one available from SysInternals http://www.sysinternals.com/Utilities/CacheSet.html (I haven't tried it lately so use it at your own risk :-))

Your comments are welcome!

SQLOS helps SQL Server to leverage hardware capabilities

slavao — Wed, 20 Jul 2005 21:50:00 GMT

I just finished an article on how SQLOS helps SQL Server to leverage hardware capabilities http://blogs.msdn.com/slavao/articles/441058.aspx. It will be interesting to hear your comments, thoughts.

Thanks a lot in advance

Be aware: FLS-Fiber Local Storage

slavao — Mon, 28 Feb 2005 19:23:00 GMT

Problem:

In the last several months there were several publications describing usage of fibers. When covering SQLOS's scheduling I will go into more details about them. Today I would like to touch on the subject on how to make your dlls/libraries more robust in the fiber environment.

As you know, when running SQL server in a fiber mode you need to be very careful when using functionality that make heavy usage of TEB, thread environment block, or TLS, thread local storage. During a fiber switch TLS and many other fields in TEB don't get switched.

You will be surprised how many libraries leverage TLS and don't work correctly with Fibers. Even older versions of MSVCRT, c-runtime, leverage TLS to allocate their per thread local data. The list of broken libraries include:

COM
RPC
Side by Side
Older versions of CRT
Windows synchronization: Critical Section & Mutex
...

So how do you protect yourself? Starting with Windows XP/2003, operating system introduced FLS, Fiber local storage: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/flssetvalue.asp. It has exactly the same semantics as TLS. However, opposite from TLS, FLS is per a fiber and it does get switched during SwitchToFiber call.

This means that if your dll will be used in fiber mode, you might want to consider using FLS instead of TLS. If a thread is not converted to a fiber, FLS is still available and there is no much overhead in using it.

If you are running XP, extended stored procedure or dll that will be loaded into SQL Server, you must use FLS. Be careful though FLS is not available on Win2K. Applications that need to run on down level platforms have to use dynamic linking - leveraging GetProcAddress API to make sure that your dll loads correctly there.

A bit of history:

FLS was finally introduced in Windows XP and 2003 because latter version of c-runtime made heavy usage of TLS for exception handling. This turned fiber mode completely unusable.

How it works:

When a thread is created it doesn't have FLS set up. Windows uses lazy initialization for FLS. When FLSAlloc gets called for the first time, OS allocates FLS for this thread. Once thread is converted to fiber, FLS gets switched during SwitchToFiber call. You can use FLSCallBack to destroy FLS entries when a fiber/thread exists or FLS gets shrunk.

I will be glad to hear your comments!

SQLOS - unleashed

slavao — Sun, 06 Feb 2005 00:38:00 GMT

With SQL Server 2005 Beta 1 and Beta 2 you might have noticed that there is no ums.dll in the bin directory. The reason is simple it is no longer exists. In SQL2000 ums.dll provided SQL Server with user mode non-preemptive scheduling. So what happened in latest version? Have we removed non-preemptive scheduling? The answer is no. Yukon continues relying on non-preemptive scheduling. As adequate research shows for RDBMS engine to meet performance and scalability requirements it needs to leverage non-preemptive scheduling.

In Yukon we took idea of non-preemptive scheduling further and have created new component, library, SQLOS, or sometimes referred as SOS. SQLOS is a user mode operating system. It includes OS pieces such as non preemptive scheduling, memory management, resource monitoring, exception handling, I/O, synchronization and hosting subsystems. Keep in mind SQLOS doesn’t provide OS abstraction layer to SQL Server. It doesn’t warp any of Windows APIs for portability purposes. Contrary it continues further binding of SQL Server to Windows by exploiting scalability and performance features of the Operating System.

The goals of SQLOS include bringing all system components together, enabling further innovation of SQL Server’s scalability and performance, providing manageability and supportability features. It exposes cohesive API to developers so that they can easily exploit features of hardware and the operating system. SQLOS models its internals to simplify developing process as for high and as for low end platforms.

SOS’s scheduling subsystem consists of scheduling nodes, schedulers, tasks, workers and system threads. A scheduling node provides an abstraction layer over group of CPUs. On NUMA systems scheduling node represents a single NUMA node. On SMP box there is one scheduling node. It is possible to change a number of nodes by editing Windows registry. We call such configuration soft NUMA. As you would expect a node also is a collection of schedulers. A scheduler is anabstraction over a CPU. Schedulers deal with tasks that run by workers that get executed by system threads. Description of SQLOS scheduling could take pages and pages so I will cover it in depth in separate post.

SQLOS’s memory management consists of memory nodes, memory clerks, caches, pools, and memory objects. A memory node is an abstraction over the memory attached to a scheduling node. A scheduling node is a proper subset of memory node. Separating scheduling and memory nodes enables us to model different hardware configurations. Every component leverages its own memory clerk to allocate large blocks of memory. Memory objects, heaps, utilize memory clerks to allocate memory. When SQL Server runs it creates a number of different memory clerks and memory objects. One could monitor them by quering SQL Server's DMVs, Dynamic Management Views. In addition to nodes, clerks and memory objects, SQLOS implements unique caching and pooling frameworks. The whole memory subsystem deserves several posts and as promised I will start covering it pretty soon.

SQLOS contains several threads to provide self monitoring and adequate resource distribution. In addition SOS exposes hosting interfaces. Hosting API significantly simplifies integration of CLR and SQL Server engines. CLR leverages SQLOS’s hosting API for all of its OS needs including thread pooling and memory management.

There is plenty of information to cover about SQLOS. One probably could write a whole book describing it. In my next post I will start going over SQLOS memory manager in deeper details. If you haven’t read my previous posts about Windows memory management it is time to do so. You need to be ready so that you can fully enjoy the next article.

Please feel free to ask any questions. And let me know your comments.

Enjoy great weekend!