NUMA and Virtual Server

Before I get started on this topic - let me pause to address the first question that many people will have - what is NUMA?  NUMA stands for 'Non-Uniform Memory Access'.  NUMA is an architecture for multiprocessor computers.  On a non-NUMA system memory is handled as a single resource, and all processors use the same pool of memory.  The problem with this is that you can get high contention rates as you add more processors.  NUMA attempts to address this contention by breaking the system up into nodes.  Each node contains its own processor(s) and a local block of memory.  The processor(s) then have low contention access to the local memory in the NUMA node, however the downside of NUMA is that if a processor needs to access memory in a different NUMA node, there is a performance penalty involved.  The exact penalty for accessing remote memory varies greatly between implementation.  On some systems (like my home system) the penalty is about 15%, but on other systems it can be as high as 600%.

Now - where does Virtual Server come into play here?  Well - by default Virtual Server will attempt to allocate processing resources and memory resources for a virtual machine on the same NUMA node.  But, this is not always possible.  To help people diagnose NUMA configuration issues - there are two possible warnings that Virtual Server can report on NUMA systems:

1. You appear to have NUMA nodes with no local memory.
   
   This warning can occur if you did not realize that you had a NUMA system and just put all the memory in one of the nodes - rather than evenly distributing the memory across the NUMA nodes.  If you see this error - you should consult your motherboard documentation and make sure that you have your memory configured appropriately (literally 'in the correct slots on the motherboard').  One potential false-positive to be aware of here is that some systems erroneously report each core on a dual-core processor as being part of a separate NUMA node - with the second core having no local memory.  If you suspect this is the case for your system - the only option you have is to a) ignore the warning and b) complain to your motherboard manufacturer to fix their BIOS to report the right information.
   
   
2. A virtual machine is currently running with its memory allocated across multiple NUMA nodes.
   
   In and of itself this is not a big deal to worry about.  If your system has a low penalty for accessing remote memory you will not see a big impact from this (I see this warning all the time on my home system - but it really does not affect me).  If, however, you are seeing unusually slow performance in a virtual machine, and this warning is being displayed - then you should do some research to find out whether your system has a high remote NUMA node access overhead - and if it does, what you can do to avoid this from happening.

A final note on NUMA - as AMD Opterons have an integrated memory controller as part of the processor - which means that any multiprocessor AMD system is inherently NUMA.

Cheers,
Ben