Slava Oks's WebLog : Be Aware

Be Aware: 4GB of VAS under WOW, does it really worth it?

slavao — Mon, 13 Mar 2006 03:12:00 GMT

By now you have heard a lot about the fact that under WOW a 32 bit process can get 4GB of VAS. I agree this is great, but I would like you to be aware of something that might hit you really hard. As it turns out enabling 4GB VAS can be problematic due...(read more)

Be aware: To Hyper or not to Hyper

slavao — Sat, 12 Nov 2005 20:48:00 GMT

Our customers observed very interesting behavior on high end Hyperthreading (HT) enabled hardware. They noticed that in some cases when high load is applied SQL Server CPU usage increases significantly but SQL Server performance degrades. Occasionally they would also see message in errorlog indicating one of the threads can't acquire a spinlock (Spinlock is a lightweight synchronization data structure. Threads keep a spinlock for short period of time. If a thread can't acquire a spinlock it will spin in a tight loop waiting for spinlock to become available). Customers also noticed that when HT is disabled under the same circumstances CPU usage increases slightly and performance is at appropriate level. What is going on here?

After some pondering and investigation we came up with the following theory. Database server has different type of threads: worker threads, threads executing clients request, and system threads that performing system tasks. An example of a system thread is a lazywriter. The responsibility of the lazywriter is to lazily scan through database buffers and push least recently used buffers out of the cache. Lazywriter usually kicks in whenever data page cache is full and server short of free data page buffers. The interesting behavior of lazywriter is that it scans large amount of memory. With Intel HT technogolgy logical processors share L1 & L2 caches. As you would guess lazywriter's behavior can potentially trash L1 & L2 caches. If a worker thread end up running on logical CPU that shares physical CPU with lazywriter its cache will be constantly trashed. It means that most of the memory accesses for the worker thread will be L1 & L2 misses. Moreover whenever the worker attempts to access hot data structure protected by spinlock it potentially might have a less chance of acquiring the spinlock which will cause the worker's spinning. This behavior will translate to high CPU utilization and significant drop in performance.

To confirm the theory I decided to create an experiment. I wrote a test program which consist of two types of threads: scan threads and lock threads. Scan threads are scanning through large amounts of memory. Lock threads constantly attempt to acquire global lock. If the theory is correct a lock thread that shares a physical CPU with a scan thread should acquire lock much less often than other threads in a given period of time. Below is the program's listing. (Disclosure: The program is for illustrative purpose only, it was written to illustrate the observed behavior, it shouldn't be treated as a final-well tested product :-))

#include <stdio.h>

#include <stdlib.h>

#include <windows.h>

#include <eh.h>

#include <wtypes.h>

#include <locale.h>

#include <process.h>

extern "C" void _mm_pause(void);

#pragma intrinsic (_mm_pause)

inline void

SmtPause ()

{

#if defined (_X86_) || defined (_AMD64_)

// PAUSE instruction (REP NOP) helps SMT scheduling

_mm_pause ();

#endif // _X86_

}

// Statistic information for threads

struct Statistics

{

int locked;

int spinned;

DWORD_PTR affinity;

};

// Type of threads the tests using

enum ThreadTypes

{

SCAN_THREAD,

LOCK_THREAD,

LAST_THREAD

};

// Entry Point

typedef void( __cdecl *ThreadEntryPoint )( void * );

// Inforamtion per thread - enables us to create threads in a common way

struct ThreadInfo

{

ThreadEntryPoint func;

LONG_PTR affinity;

};

// Global constants

// Number of CPUs

const int MaxNumberOfCPUs = 64;

// Block of memory to scan by scan thread

char MemoryBlock [1024 *1024];

// Stat information: number of threads is limmited by number of CPU,

// that is the purpose of the test

volatile Statistics Stats [MaxNumberOfCPUs];

// Number of threads running

volatile long NumberOfThreadsRunning = 0;

// "Spinlock" that all lock threads attempt to acquire

volatile long Lock = 0;

//-----------------------------------------------------------------------------

// Function: scanThread

// Description:

// Attempt to invalidate caches

// Notes:

// 1. Function changes 1MB of contiguous vm memory which appear

// to be enought invalidate L1&L2 caches

void __cdecl

scanThread (void* param)

{

char i = 0;

ULONG_PTR threadAffinity =(DWORD_PTR) param;

long threadId = InterlockedIncrement (&NumberOfThreadsRunning) -1;

SetThreadAffinityMask (GetCurrentThread (),threadAffinity);

// Initialize affinity

Stats [threadId].affinity = threadAffinity;

while (i >= 0)

{

// Memset memory: quick and dirty scan

memset (MemoryBlock, 0, sizeof (MemoryBlock));

Stats [threadId].spinned++;

}

return;

}

//-----------------------------------------------------------------------------

// Function: lockThread

// Description:

// Acquires and releases a lock in a loop

// Notes:

void __cdecl

lockThread (void* param)

{

ULONG_PTR threadAffinity =(DWORD_PTR) param;

long threadId = InterlockedIncrement (&NumberOfThreadsRunning)-1;

SetThreadAffinityMask (GetCurrentThread (), threadAffinity);

// Initialize affinity

Stats [threadId].affinity = threadAffinity;

while (1)

{

if (Lock == 0 && InterlockedExchange (&Lock, 1) == 0)

{

// Increment number of times we acquired lock

Stats [threadId].locked ++;

Lock = 0;

}

else

{

Stats [threadId].spinned++;

// Be HT friendly

SmtPause ();

continue;

}

return;

}

//-----------------------------------------------------------------------------

// Function:main

// Description:

// Creates lock and scan threads, reporting stats every 10sec

// Notes:

int __cdecl

main (

int argc,

char* argv [])

{

LONG_PTR threadId = 0;

ThreadInfo threadInfo [LAST_THREAD];

DWORD_PTR processAffinity;

DWORD_PTR threadAffinity;

DWORD_PTR systemAffinity;

int counter = 0;

int numberOfThreads = 0;

// Retrive process affinity

if (GetProcessAffinityMask (GetCurrentProcess (),

&processAffinity,

&systemAffinity) == FALSE)

{

printf ("GetProcessAffinityMask failed - exiting\n");

exit (-1);

}

// Initialize threads Information: First default and then check

// if client passed anything in

// Allocate all but one CPUs for threads running locking code

threadInfo [LOCK_THREAD].func = lockThread;

threadInfo [LOCK_THREAD].affinity = processAffinity & processAffinity-1;

// Allocate one CPU for scan thread

threadInfo [SCAN_THREAD].func = scanThread;

threadInfo [SCAN_THREAD].affinity =threadInfo [LOCK_THREAD].affinity ^ processAffinity;

// Read command line arguments: in simple way

for (counter = 0 ; counter< argc; counter++)

{

switch (*(argv [counter] + 1))

{

case 's':

case 'S':

{

// Retrieve affinity mask for scan threads

threadInfo [SCAN_THREAD].affinity =

(DWORD_PTR) _atoi64 ((argv [counter] + 2));

break;

}

case 'l':

case 'L':

{

// Retrieve affinity mask for lock threads

threadInfo [LOCK_THREAD].affinity =

(DWORD_PTR) _atoi64 ((argv [counter] + 2));

break;

}

case '?':

{

printf ("Usage: test -sn -lm \n"

"Example: test.exe -s8 -l7 (runs 1 scanning thread and 3 locking threads)\n"

"-sn is affinity of threads performing memory scan \n"

"-sm is affinity of threads acquiring lock");

exit (0);

}

default:

{

break;

}

// Start threads of all types based on supplied affinities

for (counter = 0; counter < LAST_THREAD; counter++)

{

while (threadInfo [counter].affinity != 0)

{

// Separate affiinity for a thread we are trying to

// create from all affinity mask

// Remember full mask

threadAffinity = threadInfo [counter].affinity;

// Turn off least significant bit

threadInfo [counter].affinity &= threadInfo [counter].affinity -1;

// Calculate the affinity bit for the thread:

// Leave one bit on - affinity bit

threadAffinity ^= threadInfo [counter].affinity;

// Use beginthread for simplicity

if (_beginthread (threadInfo [counter].func,

NULL,

(void*) threadAffinity) == -1)

{

printf ("Failed to create thread - exiting\n");

exit (-1);

}

numberOfThreads++;

}

// Loop and report stats periodically

while (1)

{

// Report every 10 sec

Sleep (10000);

for (threadId=0; threadId < numberOfThreads; threadId++)

{

printf ("Thread Id = %d\t, Thread Affinity = %p\t, Locked = %d\t, Spin = %d\t\n ",

threadId,

Stats [threadId].affinity,

Stats [threadId].locked,

Stats [threadId].spinned);

// Reset run time data

Stats [threadId].locked = 0;

Stats [threadId].spinned =0;

}

return 0;

}

As you can see one can specify an affinity for scan and lock threads at start up of the program.

I ran sever experiments on my dev box which has 2 physical and 4 logical CPUs. CPU 1 (Affinity 0x1) & 3 (Affinity 0x4) and 2 (Affinity 0x2) & 4 (Affinity 0x8) share physical CPUs respectively. The box also has Windows 2003 Server SP1 on it. Below are the results:

test.exe -s8 -l7

Thread Id = 0 , Thread Affinity = 00000008 , Locked = 0 , Spin = 9035

Thread Id = 1 , Thread Affinity = 00000001 , Locked = 9290331 , Spin = 10373717

Thread Id = 2 , Thread Affinity = 00000002 , Locked = 676397 , Spin = 40879794

Thread Id = 3 , Thread Affinity = 00000004 , Locked = 10274030 , Spin = 42433521

test.exe -s4 -l11

Thread Id = 0 , Thread Affinity = 00000004 , Locked = 0 , Spin = 8943

Thread Id = 1 , Thread Affinity = 00000001 , Locked = 741166 , Spin = 11421139

Thread Id = 2 , Thread Affinity = 00000002 , Locked = 10747824 , Spin = 10024252

Thread Id = 3 , Thread Affinity = 00000008 , Locked = 10210221 , Spin = 13988134

test.exe -s1 -l14

Thread Id = 0 , Thread Affinity = 00000001 , Locked = 0 , Spin = 8472

Thread Id = 1 , Thread Affinity = 00000002 , Locked = 10550011 , Spin = 20156206

Thread Id = 2 , Thread Affinity = 00000004 , Locked = 722554 , Spin = 11211074

Thread Id = 3 , Thread Affinity = 00000008 , Locked = 11182506 , Spin = 25376166

test.exe -s2 -l13

Thread Id = 0 , Thread Affinity = 00000002 , Locked = 0 , Spin = 8585

Thread Id = 1 , Thread Affinity = 00000001 , Locked = 9900899 , Spin = 12267885

Thread Id = 2 , Thread Affinity = 00000004 , Locked = 8984095 , Spin = 11909080

Thread Id = 3 , Thread Affinity = 00000008 , Locked = 1297138 , Spin = 28199769

Great! The experiment confirms the theory. So does it mean you have to disable HT when using SQL Server? The answer is it really depends on the load and hardware you are using.

You have to test your application with HT on and off under heavy loads to understand HT's implications.

Keep in mind that not only lazywriter thread can cause slowdown but any thread that performs large memory scan - for example a worker thread that scans large amount of data might be a culprit as well.

For some customer applications when disabling HT we saw 10% increase in performance. So make sure that you do your home work before you decide to hyper on not to hyper :-)

Hope this information was useful. Let me know if you have any questions!

Be Aware: VirtualAlloc with MEM_RESERVE can fail even if you have plenty of VAS inside of the process

slavao — Fri, 03 Jun 2005 19:04:00 GMT

Many developers assume that if you have plenty of free VAS in your process VirtualAlloc (http://msdn.microsoft.com/library/default.asp?url=/library/en-us/memory/base/virtualalloc.asp) and VirtualAllocEx (http://msdn.microsoft.com/library/default.asp?url=/library/en-us/memory/base/virtualallocex.asp) calls with MEM_RESERVE parameter can't fail. This is not actually true. Remember that when allocating VAS region OS creates VAD (Virtual Address Descriptor). VAD manages VAS region, for more info read http://blogs.msdn.com/slavao/archive/2005/01/27/361678.aspx. When system is low on physical memory which includes RAM and swap file, VAD's allocation might fail. The failure will cause VirtualAlloc* call to fail. It means that if system low on physical memory VirtualAlloc* could fail. This becomes very important on 64 bit platforms. When running on 64 bit, developers might assume that reservation never fails - which is wrong.

Hope it was useful.

Be Aware: Using AWE, locked pages in memory, on 64 bit

slavao — Fri, 29 Apr 2005 15:23:00 GMT

We have already talked about Windows AWE mechanism on 32 bit and how SQL Server utilizes it. Today I would like to go over AWE & related mechanism on 64 bit platforms.

To some people it comes as a surprise that AWE mechanism is still present and actually could be useful on 64 bit platforms. As you remember the mechanism consists of two parts allocating physical memory and mapping it to the given process's VAS. The advantage of allocation mechanism is that once physical memory is allocated operating system can't reclaim it until either the process is terminated or the process frees memory back to the OS. This feature allows an application to control and even avoid paging altogether. Advantage of mapping/unmapping mechanism is that the same physical page could be mapped into different VAS's regions. As you imaging unmapping is not necessary on 64 bit platforms since we have enough VAS to accommodate all of existing physical memory.

From Operating System theory, OS implements a page table entry, PTE, to describe a mapping of a page in VAS to physical page. Internally physical page is described by page frame number, PFN. Given PFN one can derive complete information about physical page it represents. For example PFN shows to which NUMA node the particular page belongs. OS has a database, collection of PFNs that it manages. If page in VAS is committed, it has PTE which might or might not point to given PFN. Conceptually, page that PTE represents can be either in memory or not, for example swapped out to disk. In the former case it is bound to a given PFN and in latter it is not. In its turn, once a physical page is bound to page in VAS, its PFN points back to PTE.

When OS commits, frees, pages out/in a given PTE or needs to derive some information about it, for example NUMA residency, it has to acquire process's working set lock - to guarantee stability of PTE to PFN binding. This lock is a rather expensive and might hurt scalability of the process. Latter versions of Windows made this lock as light as possible but avoiding still will benefit application's scalability..

When allocating physical pages utilizing AWE mechanism we are given a set of PFN entries directly from PFN database - remember that you should not manipulate or modify set of entries you get back nor can you rely on values you get back. OS is required to take a PFN database lock when allocating PFN entries. Using AWE map mechanism you can map allocated PFN entries to the process's VAS. When mapping occurs PTEs are allocated, bound to PFNs and marked as locked. In this case OS needs to acquire process's working set lock only ones. When mapping regular pages, OS does it on demand and hence will have to acquire both working set and PFN database lock for every page. Since pages are locked in memory, OS will ignore these PTEs during paging process.

On 64 bit platforms it is better to refer to such pages as locked pages - please don't confuse them with pages locked through VirtualLock API. As described above locked pages have two major properties - they are not considered for paging by OS and during allocation they acquire both working set and PFN database lock only ones.

The first property has implicit implication on high end hardware such as NUMA. It provides explicit memory residency. Remember that OS commits a page on demand. To allocate physical memory, it will use a node on which a thread touching memory is running. Latter on, the page can be swapped out by OS. Next time it will be brought up into memory, OS will again allocate physical page from the node a thread touching memory is running on. In this case a node could be completely different from original one. Such behavior makes hard for applications to rely on page's NUMA residency. Locked pages solve this problem by removing themselves from paging altogether. Moreover Windows 2003 SP1 introduced a new API - QueryWorkingSetEx. It allows to query extended information about PTE's PFN. In order to find out real page residency you should use this API. When pages are locked you need to it only ones. Otherwise you will have to do it periodically since residency of the page can actually change.

The second property - taking both working set and PFN's database lock only ones enables applications to perform faster and better scalable ramp up.

On NUMA SQL Server' Buffer Pool marks each allocated page with its node residency. It leverages QueryWorkingSetEx to accomplish it. Once page is allocated it calls the API to find out page residency. It does it only once. Therefore enabling locked pages for SQL server on 64 bit platform will improve SQL Server ramp up time and will improve performance & scalability over longer period of time. When running SQL Server with locked pages enabled you shouldn't be worried about overall system performance due to memory starvation - SQL Server participates in OS's paging mechanism by listening on OS's memory notification API's and shrinks its working set accordingly.

Let us summarize now - on 64 bit platform, locked pages, awe mechanism, enable better application's scalability and performance both during ramp up time and over long period of time. However, keep in mind that an application is still required to implement a way of responding to memory pressure to avoid starving the whole system for memory.

Let me know your comments!

Be Aware: IA64 Stack Size

slavao — Sat, 19 Mar 2005 20:35:00 GMT

Stack subject in Windows OS is fascinating. There are so many interesting technical problems surrounding it. The moment you think you fully understand everything about stack you suddenly discover yet another mystery. And so it goes.

In the last couple of weeks I have been approached by our testers and support engineers asking about for how many threads they need to configure SQL Server on IA64 platform with 4GB of RAM. For example they asked is it ok to set number of threads to be 1024?

You probably know that SQL Server commits its stacks upfront. The reason we do it is to avoid unexplained server death - if there is no physical memory and machine is out of swap file, a process can disappear without leaving any traces. This will happen when a process will try to grow its stack. As any respectful server product SQL Server can't afford such behavior.

SQL Server on IA64 is configured to use 2MB of stack size. You can verify it by looking at SQL Server image header. Does it really consume that much of committed memory per thread? The answer is no. On IA64 there are two stacks - regular stack and backing store. Backing store is a spill space for cpu register engine. It is used by cpu to spill registers to memory when it no longer has registers available.

When a process specifies committed stack size to be 2MB, Windows commits 2MB for regular stack and 2MB for backing store. Consequently each thread will commit 4MB of RAM (If you only specify reservation Windows will reserve that much but not commit). Fig1 shows what the thread stack really looks like on IA64.

----------------------0x00000000000200000

| 2MB

| Stack

| / \

| |

----------------------0x00000000000400000

| 2MB

| Backing store

| |

| \ /

----------------------0x00000000000600000

Fig 1.

I would like to emphasize couple of interesting points. First Windows reserves a contiguous VAS region of size of 4MB for both stack and backing store. Second it divides this region in two so that the stack and the backing store grow in different directions.

Now we are in the position to answer the original question of how much physical memory 1024 threads will consume. The answer is 4MB*1024= 4GB. Considering that max server memory in SQL Server only reflects size of Buffer Pool and usually it is configured to be close to amount of physical memory on the box, this configuration might cause SQL server to crawl. This comes as a real surprise to many people. Max amount of threads SQL needs to be configured for does depend on the type of application running on top of server but in this case it definitely needs to be below 1024. Even 512 number of threads could be too much.

This post will be incomplete if I didn't somehow mention stack overflow. Backing store makes IA64 stack handling to be different from other platforms. Stack overflow is not an exception in this case - OS needs to handle stack overflow both for regular stack and backing store. Surprise! For you it means that you have to be really careful when recovering from stack overflow especially when you diced to restore guard page by hand. But this is a whole different story on its own.

I hope you learned something from this post. Comments are welcomed!

Have a great weekend!

Be aware: FLS-Fiber Local Storage

slavao — Mon, 28 Feb 2005 19:23:00 GMT

Problem:

In the last several months there were several publications describing usage of fibers. When covering SQLOS's scheduling I will go into more details about them. Today I would like to touch on the subject on how to make your dlls/libraries more robust in the fiber environment.

As you know, when running SQL server in a fiber mode you need to be very careful when using functionality that make heavy usage of TEB, thread environment block, or TLS, thread local storage. During a fiber switch TLS and many other fields in TEB don't get switched.

You will be surprised how many libraries leverage TLS and don't work correctly with Fibers. Even older versions of MSVCRT, c-runtime, leverage TLS to allocate their per thread local data. The list of broken libraries include:

COM
RPC
Side by Side
Older versions of CRT
Windows synchronization: Critical Section & Mutex
...

So how do you protect yourself? Starting with Windows XP/2003, operating system introduced FLS, Fiber local storage: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/flssetvalue.asp. It has exactly the same semantics as TLS. However, opposite from TLS, FLS is per a fiber and it does get switched during SwitchToFiber call.

This means that if your dll will be used in fiber mode, you might want to consider using FLS instead of TLS. If a thread is not converted to a fiber, FLS is still available and there is no much overhead in using it.

If you are running XP, extended stored procedure or dll that will be loaded into SQL Server, you must use FLS. Be careful though FLS is not available on Win2K. Applications that need to run on down level platforms have to use dynamic linking - leveraging GetProcAddress API to make sure that your dll loads correctly there.

A bit of history:

FLS was finally introduced in Windows XP and 2003 because latter version of c-runtime made heavy usage of TLS for exception handling. This turned fiber mode completely unusable.

How it works:

When a thread is created it doesn't have FLS set up. Windows uses lazy initialization for FLS. When FLSAlloc gets called for the first time, OS allocates FLS for this thread. Once thread is converted to fiber, FLS gets switched during SwitchToFiber call. You can use FLSCallBack to destroy FLS entries when a fiber/thread exists or FLS gets shrunk.

I will be glad to hear your comments!

Be aware: a new section in my blog

slavao — Tue, 22 Feb 2005 09:12:00 GMT

Periodically, I bump into unusual behavior, features, of compiler, operating system, strange design and coding decisions made by someone else that makes me wonder. After giving it some thought, I often come to the conclusion that behavior or observed specifics are justifiable. However if I knew it upfront, It would have helped me to avoid unnecessary mistakes and complexity.

Today I would like to open new section in my blog: Be Aware. In this section I would like to spend some time on this exact subject: strange behaviors and unexpected features. Sometimes these type of things are fascinating, sometimes just interesting and sometimes are just naïve. My goal is to point them out so that when you make your own decisions you will have more information to rely on. My goal is not to uncover bugs or start some kind of religious debates - I really don't like them. I strongly believe that problem drives a solution. What I mean by this is that something outrages in one place will be fully applicable in the other.

So let's begin :-).

The topic of variable's placement have been covered to full extend by now. Some developers prefer to declare all variables upfront some in the scope where variables get actually used. Personally, I have heard cons and pros for both sides and kind of agree with both. I think there is no black and white. In some cases you do have to declare a variable inside of the given scope. Recently I have observed interesting debugger's behavior while debugging dumps. As all of you know debugger usually has difficulties to show local variables for retail, optimized build. This is kind of expected in many cases. Before you can rely on a value, you have to confirm that the value is really correct. In windbg I usually use dv command to output local variables

> dv -V

Using the command with -V lists the actual placement of the variable on the stack. What I noticed is that windbg wasn't displaying variables declared in local scopes at all even if instruction pointer was inside of the scope. It just doesn't list them! So I usually end up chasing those variables by disassembling the code and following stack manipulations. (Knowing original stack assignment in this cases would really helped) The process works 100% of the time, but I would have saved hours if the code declared variables in the beginning of the function. At least the debugger would gave me a clue where the variable resides on the stack.

Probably the point here is that avoiding unnecessary complexity could enable tools to do better job and make everyone life easier. Remember tools, as any software, well tested for the common case. So next time when declaring a variable in an internal scope just stop for a second and think: Is it really necessary?

I have more stories around complexity but I will keep them until next time :-).