Lessons from the test lab: investigating a pleasant surprise

Just another day in the perf lab

Rico Mariani's Performance Tidbits — Thu, 19 Jun 2008 00:55:22 GMT

Even though I've been doing general architecture work on Visual Studio for nearly a year now, my friends

re: Lessons from the test lab: investigating a pleasant surprise

Antonio D — Thu, 19 Jun 2008 04:48:48 GMT

To explain the memory bandwidth difference...

The pentium 4 (830D) was a more memory hungry architecture. Risking a gross oversimplification I would say that the 830 D agressively transfers memory in its cache, even if it ends up not doing it. So the bandwidth measured by a benchmark in a best case scenario could be much higher than the actual bandwith that your software can use.

About the amount of cache used by a thread.

I was under the impression that Windows scheduled threads on different cores. i.e. thread x is not guaranteed and is not going to run always on the first core of the processor. So if you have two concurrent threads, they are probably running both on both (or all four) cores. In this case the amount of cache used by each thread would be undetermined (except by further analysis).

My understanding is that the part where you say that thread gets more of the cache when it needs it is is true regardless of the number of cores or wether these cores are sharing a cache or not. What am I missing?

re: Lessons from the test lab: investigating a pleasant surprise

MarkBFriedman — Thu, 19 Jun 2008 08:28:47 GMT

Antonio:

It looks like we should have been a little clearer about what we meant when we used the word "thread." Sorry about that. (Reminds me of the famous words of a former US President and semiotician, "It depends on what the meaning of the word "is" is.")

From the standpoint of the OS, the thread is the dispatchable unit. From the standpoint of the CPU, a thread is any set of executing instructions that aren't executing an Idle loop. There are software and hardware guys collaborating on this post. (It may be unusual, but they do get along -- most days.) And while we knew what we meant, it seems we used "thread" without clearly distinguishing the two meanings and contexts.

Knowing the author and his tendencies, my guess is that the footnote about threads sharing the cache was written from the hardware perpective.

On the new lab machines, there is a dedicated L1 cache for each processor core, and a shared L2 cache that each processor core can access. The L2 cache is dynamically allocated. If CPU A is idle and CPU B is cranking, CPU B is capable of allocating the entire L2 cache. (If you don't expect the CPUs sharing the cache to all be cranking all of the time, this is probably a good approach.) I hope that clarifies the point.

Of course, I am a software guy, so, from the Windows point of view, let me also try to clarify your thread dispatching question:

It is true that "thread x is not guaranteed and is not going to run always on the first core of the processor." Having said that, however, the statement that follows isn't entirely true: "So if you have two concurrent threads, they are probably running both on both (or all four) cores."

Yes and no. On a symmetric multiprocessor (SMP), a thread by default tends to be a bit sticky to the processor it was last dispatched on. This is called "soft affinity" and is done to increase the probablity that a cache warm start will occur. This stickiness is especially noticeable when the processors are lightly loaded. The stickiness is also prominent in the WinSat benchmarks described here that run single threaded and were run in insolation.

But, in general, you are correct and you often observe threads switching back and forth between available processors. Because thread scheduling is priority-based with preemptive scheduling, and User mode threads are typically subject to dynamic adjustments, once the machine is loaded, threads will usually wander (somewhat randomly) from CPU to CPU.

The SMP soft affinity scheduling algorithm is roughly as follows: A waiting thread that transitions to the Ready state has an "ideal processor" where it will run if that processor is currently idle or running a lower priority thread. If the ideal (i.e., last) processor is busy or running a higher priority thread, but another processor is idle or running a lower priority thread, the ready thread will be scheduled there. This is the preemptive scheduling bit -- the highest priority Ready threads are always dispatched.

You will find more details in my Windows Server 2003 Resource Kit "Performance Guide" book: the priority scheme, hard processor affinity, etc. The ntttcp program discussed in my recent "Mainstream NUMA and the TCP/IP stack" post used hard processor affinity, for example, to ensure that all network processing was confined to a single CPU. Which was why I only showed what was happening on the one CPU. Hard affinity is the exception, though, not the rule.

Once you move to a NUMA architecture -- see my earlier blog posts on this subject, like it or not, NUMA is in your future if you are running server-class machines -- the thread scheduling scheme gets node-oriented. (Physical memory allocations are also node-oriented on NUMA machines.) Once scheduled on a node, a thread is likely to continue to be scheduled to run on one of that node's CPUs. (Subject to availability, similar to the SMP case.)

This NUMA node-oriented soft affinity scheme works pretty well when the L2 cache is a resource that is shared by all the processor cores on the socket. In today's multi-core machines, so long as the thread is re-dispatched to a CPU on the same socket (or node) where it last ran, the thread will likely benefit from an L2 cache warm start. But for an L1 cache warm start, the thread still has to be dispatched on its ideal (and still preferred) processor since that resource is dedicated.

This description of the the behavior of the Windows Scheduler is also worth my mentioning here because of its significance to my earlier "Mainstream NUMA and the TCP/IP stack" posting. In the next part of "Mainstream NUMA" post, which I hope to have ready in another week or so, I will try to make this connection explicit.

So, thanks for keeping us honest and stay tuned!

-- Mark

re: Lessons from the test lab: investigating a pleasant surprise

Antonio D — Thu, 19 Jun 2008 15:19:03 GMT

Thank you for the answer. I feel enlightened now!

Especially about soft affinity. I kind of always worried about that.

Performance analysis of multi-core systems

Jonathan Hardwick — Thu, 19 Jun 2008 23:30:48 GMT

One of our main roles in DevDiv Performance Engineering is to help other teams with performance investigations

re: Lessons from the test lab: investigating a pleasant surprise

Mohit Nanda — Wed, 02 Jul 2008 09:57:43 GMT

Thanks Mark about the enlightning details about 'Soft Affinity', and the reference to your Win2003 Performance Engineering Handbook was also useful.

Looking up to next part of "Mainstream NUMA" post.

Visual Studio 2010 Hardware Requirements

(Semi) Official Developer Division Performance Engineering blog — Wed, 24 Dec 2008 11:00:07 GMT

Soma’s been talking about the upcoming Visual Studio 2010 release on his blog , which means I’m starting

Are we taking advantage of Parallelism?

Developer Division Performance Engineering blog — Sun, 03 May 2009 01:41:23 GMT

Recently, a colleague of mine, Mark Friedman, posted a blog titled “ Parallel Scalability Isn’t Child’s