Developer Division Performance Engineering blog : Performance Engineering

Are we taking advantage of Parallelism?

Sunny Egbo — Sun, 03 May 2009 01:38:00 GMT

Recently, a colleague of mine, Mark Friedman, posted a blog titled “Parallel Scalability Isn’t Child’s Play” in which he reviewed the merits of Amdahl Law vs. Gunther’s Law for determining the practical limits to parallelization. I would not argue with the premise of Mark’s blog that Parallelism is not child’s play. However, I do have alternate views of the use of Amdahl Law and Gunther’s Law that I posted on his blog. I think my views and comments on Mark’s blog warrant another blog to fully explain.

Speaking of child’s play: my 10-year old son recently made a two-part movie titled “the Way” and “the Way Back” complete with a full storyline, multiple sound tracks and narrations. He put these movies together with only the help of his eight-year old sister, using sample movie clips and stock photographs he found on his computer hard drive. He asked me for help getting his two masterpieces onto a DVD capable of playing on the average home DVD player. Also, he asked about the length of a typical movie playing in movie theaters around the U.S. (approximately 2 hours) and how much these movies cost at the movie theater (approximately $12 for adults and $8 for children, minus the popcorn). Based on my answers, he determined that he will charge 25 cents for people to watch his movies, because he wanted everyone to attend. I wanted to ask him how much he would charge someone who decided to watch only one of the clips. However, I didn’t because I did not want to lose a price haggling war with a 10-year old. Besides, it would be terrible if you cannot find your way back.

In any case, his movies were quite impressive. The most technologically savvy thing I did as a 10-year old kid was to build a telephone line with tomato soup cans and a string. Movie making was out of reach for me; but now it is child’s play.

Today, parallelism is not child’s play. However, I hold out hope that in the future the typical computer program would be written with parallelism in mind. Is parallelism ever going to be child’s play in the future the way movie making is today?

Parallelism exists everywhere: instruction level, memory level, loop level and task level parallelism, etc. Also, parallelism has been with us for quite some time now. For the past several decades, hardware engineers have quietly been busy solving problems in parallel to improve processor and system level performance. However, for the past four or more years, hardware designers have encountered the twin brick walls created by memory speed and power. These walls have forced CPU architects and hardware designers to go multi-core in a major way. The doubling of the CPU frequency every 18 months, that was true for many decades, are no longer practical and have come to an abrupt end. Although, hardware performance continues to improve as my colleagues and I pointed out in our blog “Investigating a Pleasant Surprise,” the pace of CPU frequency increases has slowed considerably. Instead, hardware designers have been doubling the number of cores available on a single CPU socket every couple of years.

To get the same level of performance that was previously possible, software engineers would now need to step up to the plate—to write software in a parallel and scalable fashion. They would need tools and frameworks that allow them to think about their problems, identify opportunities for parallelism and to analyze their solutions correctly and efficiently.

I am a big fan of Amdahl Law as an analysis framework. However, I do not subscribe to the narrow view that Amdahl’s Law applies only to parallelism, as most people who write about it seem to imply. I prefer the broader treatment of the Law by Hennessy and Patterson in their famous book “Computer Architecture: A Quantitative Approach”—where Amdahl’s Law is used to estimate the opportunities between competing designs. Amdahl’s Law is very powerful for showing the areas that will likely yield the most fruitful performance gains. In my performance design, tuning and optimization work, I use Amdahl Law for prioritizing the areas of opportunities to focus my efforts to gain performance.

Amdahl’s Law is not the limit to either absolute performance or parallelism as many authors seem to suggest. Gunther’s and Gustafson’s Laws are helpful for putting Amdahl’s Law in perspective. However, like Amdahl’s Law, these laws are not fundamental limits. The use of these three laws to estimate the level of parallelism that is possible is very flawed. Specifically, the use of these laws as fundamental limits can obfuscate the level of parallelism and performance inherent in typical computing problems. These laws gloss over a number of important points and practical aspects of obtaining parallelism in general purpose computing, including that:

1. Many user tasks are non-monolithic and can be solved in a distributed fashion. Background tasks (e.g., virus scans) that often block single processor execution can now be done in a way that improves user experiences. The key is to identify unnecessary dependencies that would allow these tasks to proceed in parallel with other tasks in a multi-core computer.

2. Some algorithms that have inefficient sequential solutions surprisingly have efficient parallel solutions. This fact should be comforting to fans of algorithms. For example, many applications require matrix multiplication, which turns out to be easily parallelizable. Although the best sequential algorithm for matrix multiplication has a time complexity of O(n^2.376), a straight-forward parallel solution has an asymptotic time complexity of O(log n) using n^2.376 processor. In other words, we can readily find a parallel solution for matrix multiplication that improves its runtime as more and more processor cores become available. Of course, you might have difficulty conceiving of n^2.376 processors in a system--as a colleague mentioned recently. However, this is just another way of saying that matrix multiplication will benefit with more and more processors.

3. Some poor sequential algorithms can be easily parallelized to execute in less time than their sequential solutions. Also, we know that some algorithms that have the best asymptotic time complexities achieve their speed by introducing data dependencies that make parallelization difficult and that the best asymptotic time complexity does not necessarily translate to the best runtime in real life. Hence, at some point, the benefit of the simpler parallelization of some poor sequential algorithms that have little data dependencies can outweigh the benefit of more efficient sequential counterparts that have data dependencies. Hence, when considering parallel solutions it is not always necessary to start with the sequential solution with the best time complexity [also, see comment about Fortune and Wylie below].

4. The real world performance of applications is not determined exclusively by the asymptotic time complexity of algorithms. Because of the increasing gap between CPU and memory speed, memory accesses are increasingly dominating the performance of applications running on modern CPUs. Although, the gap can be mitigated with large caches, every cache miss takes hundreds of CPU cycles to complete. Even a modest overlap in these memory accesses (Memory Level Parallelism) can improve application performance in noticeable ways.

Over the years, there have been efforts to classify computationally intractable problems. Many decision problems (i.e., Yes/No) and their optimization counterparts have been categorized into NP-complete and NP-Hard sets respectively. The Travelling Salesman (TSP), Online Bin-Packing and 3-Dimensional Matching problems are three famous examples of NP-Complete problems. In a similar fashion, problems that are difficult to parallelize have been categorized into the P-Complete set or the set of problems that are known to be inherently sequential. As you can imagine sorting is not P-Complete. Likewise, Matrix Multiplication is not in the P-Complete set. Processor scheduling can be done in O(log n) time units using n processors—so, it is not P-Complete either. In an ultimate twist of irony, many NP-Hard problems have heuristic solutions that can be executed in parallel to approximate the real solutions. Hence, the natural inclination to think that NP-Complete problems cannot be parallelized is not borne out in practice.

As it turns out, the real limit to parallelism seem not to be defined by Amdahl’s Law, Gunther’s Law, Gustafson’s Law or NP-Completeness, but by the P-Complete set. It appears that parallelizable problems are related to the asymptotic space complexity of their sequential solutions. According to the Fortune and Wylie’s Parallel Processing Thesis, any problem that can be solved with a poly-logarithmic space complexity can be parallelized efficiently. Because of the time space trade-off of algorithms, this implies that the sequential algorithm that achieves this space complexity is not necessarily the algorithm with the best asymptotic time complexity.

In any case, because one can evaluate problems on multiple levels beyond algorithms (e.g., at the instruction, memory and data access, loop and task levels), the set of problems that can be parallelized appears to be quite large. The question is how to identify and take advantage of the parallelization opportunities that may be inherently available and to do so in an efficient and scalable manner. How can we parallelize loops? How do we overlap high latency activities such as accesses to physical memory or I/O to amortize the cost of those activities? How do we minimize synchronizations? How do we partition tasks to eliminate bottlenecks from the critical paths? How do we dispatch work efficiently to improve efficient system utilization, improve throughput and improve latency? What areas of our application can benefit from what sets of efforts? These are some of the questions that allow for scalable designs.

Today, the tools to identify parallelism and scalability opportunities are very limited. The programming languages that allow programmers to express parallelism in a natural way are completely lacking. The tools to analyze and troubleshoot parallel implementations are limited as well. Debugging parallel implementation is particularly hard. However, I suspect that with some industry focus and incremental progress, we could continue to make parallelism accessible to average programmers in a few years. However, we are many years away.

What are some of the fundamental limits preventing such tools to be built? Like Mark said on his blog, achieving improved scalability using parallel programming techniques is certainly very challenging. But, can parallel programming be made less challenging with intuitive tools that expose parallel solutions in a natural way and allow programmers to exploit them? Can programming languages and tools improve to a point where a typical 10-year old will be able to write a parallel program as easily as they can put together a multi-track movie today?

Sunny Egbo

PDC2008 preConference Workshop

MarkBFriedman — Wed, 22 Oct 2008 18:36:00 GMT

Over the past several weeks, I have been working overtime developing a presentation on web application performance to be given at the upcoming Professional Developer’s Conference (PDC), which is next week in Los Angeles. This is partly why I have been remiss about blogging this month. At least, that is my excuse, and I am sticking to it.

The presentation is entitled “Performance by design using the .NET Framework” and I am presenting jointly with two colleagues in the Developer Division, Rico Mariani and Vance Morrison. It is one of ten PreConference sessions that are scheduled to run all day on Sunday. My portion of the session is an extended discussion of optimization & scaling strategies for web applications. The scope encompasses ASP.NET, AJAX, Silverlight, WPF & WCF. Information about the upcoming event is here.

I have attended several PDCs in the past as a customer, and found them to be amazing events. Before the days of widespread blogging, the “Ask the Experts” sessions at the PDC were often the only way to get an authoritative answer to your question. The actual Conference sessions emphasize imminently arriving technology and future directions, aimed at the professional developer who needs to be able to anticipate and plan. The technical sessions run the gamut from Windows 7, the Windows Live Cloud computing initiatives, IE8, Surface and Windows for Workflow. There will be previews of the next version of the .NET Framework, Visual Studio, and the Visual Studio Team System.

This is the first time I will be on the other side of the podium for the event. In our preCon session, Rico, Vance and I will focus on facilities available in the Framework today, including the best practices and tools we recommend to help you design & build an application that meets its performance and scalability requirements. The intended audience is experienced .NET developers.

If you are reading this blog & coming to my session, be sure to say hello. I’d like to get the chance to meet you in person.

-- Mark Friedman

Mainstream NUMA and the TCP/IP stack: Final Thoughts

MarkBFriedman — Fri, 19 Sep 2008 00:18:00 GMT

This is a continuation of Part IV of this article posted here.

Note that a final version of a white paper tying this series of five blog entries together (and a Powerpoint presentation on the subject) are attached.

For many years, the effort to improve network performance on Windows and other platforms focused on reducing the host processing requirements associated with the need to service frequent interrupts from the NIC. In the many-core era where the clock speeds of processors are constrained by power considerations, this strategy is inadequate to the growing host processing requirements that accompany high-speed networking. It is necessary to augment technologies like interrupt moderation and TCP Offload Engine that improve the efficiency of network I/O with an approach that allows TCP/IP Receive packets to be processed in parallel across multiple CPUs. Together, MSI-X and RSS are technologies that enable host processing of TCP/IP packets to scale in the many-core world, albeit not without some compromises with the prevailing model of networking using isolated, layered components.

Using MSI-X and RSS, for example, the Intel 82598 10 Gigabit Ethernet Controller mentioned earlier can be mapped to a maximum of 16 processor cores that could then be devoted to networking I/O interrupt handling. Capacity-wise, this is still not sufficient processing capacity to handle the theoretical maximum load equation 3 predicts for a 10 Gb Ethernet card, but it does represent a substantial scalability improvement.

With this understanding of what MSI-X and RSS accomplishes, let’s return for a moment to our NUMA server machine shown in Figure 6 below.

With MSI-X and Receive-Side Scaling, CPU 0 on node A and CPU 1 on node B are both enabled for processing network interrupts. Since RSS schedules the NDIS DPC to run on the same processor as the ISR, even at moderate networking loads, CPU 0 and 1 for all practical purposes become dedicated to the processing of high priority networking interrupts.

Numerous economies of scale accrue using this approach. The same RSS process that sends all Receive packets from a single TCP connection to a specific CPU for processing improves the efficiency of that processing. The instruction execution rate of the TCP/IP protocol stack is enhanced significantly through this scheduling mechanism that enforces localization. Ultimately, TCP/IP application data buffers need to be allocated from local node memory and processed by threads confined to that node. Recently used data and instructions that networking ISRs and DPCs issue tend to reside in the dedicated cache (or caches) associated with the processor devoted to network I/O. Or, at the very least, they migrate to the last level cache that is shared by all the processors on the same NUMA node.

Ultimately, of course, the TCP layer hands data from the network I/O to an application layer that is ready to receive and process it. The implications of RSS for the application threads that process TCP receive packets and build responses for TCP/IP to send back to network clients ought to be obvious, but I will spell them out anyway. For optimal performance, these application processing threads also need to be directed to run on the same NUMA node where the TCP Receive packet was processed. This localization of the application’s threads should, of course, be subject to other load balancing considerations to prevent the ideal node from becoming severely over-committed while other CPUs on other nodes are idling or under-utilized. The performance penalty for an application thread that must run on a different node than the one that processed the original TCP/IP Receive packet is considerable because it must access the data payload of the request remotely. Networked applications need to understand these performance and capacity considerations and schedule their threads accordingly to balance the work across NUMA nodes optimally.

Consider the ASP.NET application threads that process incoming HTTP Requests and generate HTTP Response messages. If the HTTP Request packet is processed by CPU 0 on node A in a NUMA machine, the Request packet payload is allocated in node A local memory. The ASP.NET application thread running in User mode that processes that incoming HTTP Request will run much more efficiently if it is scheduled to run on one of the other processors on node A, where it can access the payload and build the Response message using local node memory.

There is currently no mechanism in Windows today for kernel mode drivers like ndis.sys and http.sys to communicate to the application layers above them and specify the NUMA node on which that packet was originally processed. Communicating that information to the application layer is another grievous violation of the principle of isolation in the network protocol stack, but it is a necessary step to improve the performance of networking applications in the many-core era where even moderately sized server machines have NUMA characteristics.

Links.

Herb Sutter, “The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software.” Dr. Dobb’s Journal, March 1, 2005. http://www.ddj.com/architect/184405990

NTttcp performance testing tool: http://www.microsoft.com/whdc/device/network/TCP_tool.mspx

Windows Performance Toolkit (WPT, aka xperf): http://msdn.microsoft.com/en-us/library/cc305218.aspx

David Kanter, “The Common System Interface: Intel's Future Interconnect,” http://www.realworldtech.com/includes/templates/articles.cfm?ArticleID=RWT082807020032&mode=print

Windows NUMA support: http://msdn.microsoft.com/en-us/library/aa363804.aspx

Intel white paper: Accelerating High-Speed Networking with Intel® I/O Acceleration Technology

Mark B. Friedman, “An Introduction to SAN Capacity Planning,” Proceedings, Computer Measurement Group, Dec. 2001.

Jeffrey Mogul’s “TCP offload is a dumb idea whose time has come,” Proceedings of the 9th conference on Hot Topics in Operating Systems - Volume 9, 2003. http://portal.acm.org/citation.cfm?id=1251059&dl=ACM&coll=portal&CFID=71988909&CFTOKEN=98964748

Dell Computer Corporation, “Boosting Data Transfer with TCP Offload Engine Technology.”

Microsoft Corporation, KB 951037, http://support.microsoft.com/kb/951037

Microsoft Corporation, Windows Driver Development Kit (DDK) documentation, http://msdn.microsoft.com/en-us/library/cc264906.aspx

Microsoft Corporation, KB 927168, http://support.microsoft.com/kb/927168

Microsoft Corporation, NDIS 6.0 Receive-Side Scaling documentation, http://msdn.microsoft.com/en-us/library/ms795609.aspx

Mainstream NUMA & the TCP/IP stack: Part 2: Programming ccNUMA machines

MarkBFriedman — Sun, 27 Jul 2008 21:02:00 GMT

This is a continuation of Part I of this article posted here.

In Part 1 of this article, we looked at the capacity issues that are driving architectural changes in the TCP/IP networking stack. While network interfaces are increasing in throughput capacity, processor speeds in the multi-core era are not keeping pace. Meanwhile, the TCP/IP protocol has grown in complexity so that host processing requirements are increasing, too. The only way for networked computers to scale in the multi-core era is to begin distributing networking I/O operations across multiple processors. Since bigger server machines rely on NUMA architectures for scalability, high speed networking is also evolving to exploit machines with NUMA architectures in an optimal fashion.

Machines with NUMA (non-uniform memory access speeds) architectures are usually large scale multiprocessors that are assembled using building blocks, or nodes, that each contain some number of CPUs, some amount of RAM, and various other peripheral connections. Nodes are often configured on separate boards, for example, or specific segments of a board. Multiple nodes are then interconnected with high speed links of some sort that permit all the memory that is configured to be available to executing programs. There are many schools of thought on what the best interconnection technology is. Some manufacturers favor tree structures, some favor directory schemes, some favor network-like routing. A key feature of the architecture is that the latency of a memory fetch depends on the physical location of the RAM being accessed. Accessing RAM attached to the local node is faster than a memory fetch to a remote location that is physically located on another node.

Within one of the new Intel Nehalem many-core microprocessor, for example, all the processor cores and their logical processors can access local memory at a uniform speed. Figure 3 is a schematic diagram depicting a 4-way Nehalem multiprocessor chip that is connected to a bank of RAM. The configuration of processors and RAM shown in Figure 3 is a building block that is used in creating a larger scale machine by connecting two or more of such nodes together.

Figure 3. A schematic diagram depicting a NUMA node showing locally-attached RAM and a multi-core socket.

A two-node NUMA server is illustrated in Figure 4, which shows a direct connection between the memory controller on node A and the memory controller on node B. This is the relatively simple case. A thread executing on node A can access any RAM location on either node, but an access to a local memory address is considerably faster. The latency to access to a remote memory location is several times slower. (Definitive timings are not available as of this writing because early versions of the hardware are just starting to become available.)

Figure 4. A two-NUMA server showing a cross-node link that is used when a thread on one node needs to access a remote memory location.

As the number of nodes increases, it is no longer feasible for every node to be directly connected to every other node, nor can each bank of RAM that is installed be accessed in a single hop. The specific technology used to link nodes may introduce additional variation in the cost of accessing remote memory. From any one node, it could take longer to access memory on some nodes than others. For instance, some nodes may be accessed in a single hop across a direct link, while other accesses may require multiple hops. Some manufacturers favor routing through a shared directory service, for example. Your mileage may vary.

Specifically, in the Intel architecture, manufacturers are supplying a cache coherent flavor of NUMA servers (ccNUMA). Cache coherence is implemented using a snooping protocol to ensure that threads executing on each NUMA node have access to the most current copy of the contents of the distributed memory. Details of the snooping protocol used in Intel ccNUMA machines are discussed here.

AMD has taken a somewhat different tack in building its multi-core processors. For communication on chip between processors, AMD uses a technology known as HyperTransport, which is a dedicated, per-processor 2-way high speed link. Multiple processors cores are then linked on the chip in a ring topology as depicted in Figure 5. The ring topology has the effect of scaling the bus bandwidth that is used as an interconnect linearly with the number of the processors. But the architecture leads to NUMA characteristics. A thread executing on CPU 0 can access a local memory location, a remote memory location that is local to CPU 1 at the cost of one hop across the HT link, or a remote memory location that is local to CPU 2 at the cost of two hops across HT links.

Figure 5. The AMD approach to multi-core processors has NUMA characteristics. A program executing on CPU 0 that accesses RAM that is local to CPU 2 requires two hops across the HyperTransport links that connect the processors in a ring.

Historically, application development for NUMA machines meant understanding the performance costs associated with accessing remote memory on a specific hardware platform. Since manufacturers employ different proprietary interconnection schemes in their multi-tiered NUMA machines, application developer are challenged to find the right balance in exploiting a specific proprietary architecture that may then limit the ability to port the application to a different platform in the future. It may be possible to connect nodes in a NUMA machine in an asymmetric configuration, for example, where the performance cost function associated with accessing different memory locations is decidedly irregular.

To scale well, a multi-threaded program running on a NUMA machine needs to be aware of the machine environment and understand which memory references are local to the node and which are remote. A thread that was running on one NUMA node that migrates to another node pays a heavy price every time it has to fetch results from remote memory locations. The difficulty programmers face when trying to develop a scalable, multi-threaded application for a NUMA architecture machine is understanding their memory usage pattern and how it maps to the NUMA topography. When NUMA considerations were confined to expensive, high-end supercomputers, the inherent complexities developers faced in programming them were considered relatively esoteric concerns. However, in the era of many-core processors, NUMA is poised to become a mainstream architecture.

In theory, it is possible to craft an optimal solution when threads and the memory they access are balanced across NUMA processing nodes. In order to achieve an optimal balancing of the machines resources without overloading any of them, programs need to understand the CPU and memory resources that individual tasks executing in parallel require and understand how to best map those resources to the topography of the machine. Then they require a suitable scheduling mechanism to achieve the desired result. Achieving an optimal balance, as a practical matter, is not easy, in the face of variability in the resources required by any of execution threads, a complication that may then require dynamic adjustments to the scheduling policy in effect.

The Windows OS is already NUMA-aware to a degree and, thus, supports a NUMA programming model. For example, once dispatched, threads have node affinity and tend to stay dispatched on an available processor within a node. Windows OS memory management is also NUMA-aware, maintaining per node allocation pools. The OS not only resists migrating threads to another node, it also tries to ensure that most memory allocated are satisfied locally using per node memory management data structures.

Windows also provides a number of NUMA-oriented APIs that applications can use to keep their threads from migrating off-node and also enable them to direct memory allocations to a specific physical processing node. For more information on the NUMA support in Windows, see the MSDN Help topic “NUMA Support.”

To help application developers deal better with the complexities of NUMA architectures in the future, the Windows NUMA support needs to evolve. One potential approach would be for the OS to attempt to calculate a performance cost function at start-up that it would then expose to driver and application programs when they start up and run. Conceivably, the OS might also need to adjust this performance cost function to response to configuration changes that occur dynamically, such as any power management event that affects memory latency. These changes would then have to be communicated to NUMA-aware drivers and applications somehow so they could adapt to changing conditions.

Continue to Part III of this article.

Mainstream NUMA and the TCP/IP stack: Part I.

MarkBFriedman — Tue, 10 Jun 2008 04:44:00 GMT

One of the intriguing aspects of the onset of the many-core processor era is the necessity of using parallel programming techniques to reap the performance benefits of this and future generations of processor chips. Instead of significantly faster processors, we are getting more of them packaged on a single chip. To build the cost-effective mid-range blade servers configured in huge server farms to drive today’s Internet-based applications, the hardware manufacturers are tying together these complex multiprocessor chips to create NUMA architecture machines. There is nothing the matter with NUMA – machines with non-uniform memory access speeds – of course, other than the fact that they introduce complex, hardware-specific programming models if you want to build applications that can harness their performance and capacity effectively. What is decidedly new is the extent to which previously esoteric NUMA architecture machines are becoming mainstream building blocks for current and future application servers. For the connected applications of the future, our ability to build programming models that help server application developers deal with complex NUMA architecture performance considerations is the singular challenge of the many-core era.

In this blog entry, I will discuss the way both these trends -- multi-core processors and mainstream NUMA architectures – come together to influence the way high speed internetworking works today on servers of various sorts that need to handle a high volume of TCP/IP traffic. These include IIS web servers, Terminal Servers, SQL Servers, Exchange servers, Office Communicator servers, and others. Profound changes were necessary in the TCP/IP networking stack in both Windows Server 2008 and the Microsoft Windows Server 2003 Scalable Networking Pack release to scale effectively on multi-processor machines. These changes are associated with a technology known as Receive-Side Scaling, or RSS. RSS has serious performance implications on the architecture of highly scalable server applications that sit atop the TCP/IP stack in connected system environments.

Let’s start by considering what is happening to the TCP/IP software stack in Windows to support high speed networking, which is depicted in Figure 1.

The Internet Protocol (IP) and the Transmission Control Program (TCP) are the standardized software layers that sit atop the networking hardware. The Ethernet protocol is the pervasive Media Access (MAC) Layer that segregates the transmission of digital bits into individual packets. Performance issues with Ethernet arise due to the relatively small size of each packet. The Maximum Transmission Unit (MTU) for standard Ethernet sized packets is 1500 bytes. Any messages that are larger than the MTU require segmentation to fit in standard sized Ethernet packets. (Segmentation on the Send side and reassembly on the Receive side are functions performed by the next higher level protocol in the stack, namely the IP layer.) Not all transmissions are maximum sized packets. For example, the Acknowledgement (ACK) packets required and frequently issued in TCP consist of 50-byte packet headers only, with no additional data payload. On average, the size of packets circulating over the Internet is actually much less than the protocol-defined MTU.

The performance problems arise because in a basic networking scheme, each packet received by the Network Interface Card (NIC) delivers a hardware interrupt to the host processor, requiring in turn some associated processing time on the host computer to service that interrupt. The TCP/IP protocol is reasonably complex, so the amount of host processing per interrupt is considerable.[1] As transmission bit rates have increased from 10 Mb to 100 Mb to 1 Gb to today’s 10 Gb NICs, the potential interrupt rate rises proportionally. The host CPU load associated with processing network interrupts is a long-standing issue in the world of high speed networking. The problem has taken on a new dimension in the many-core era because the network interface continues to get faster, but processor speeds are no longer keeping pace.

A back-of-the envelope calculation to figure out how many interrupts/sec a host computer with a 10 Gb Ethernet card potentially needs to handle should illustrate the scope of the problem. The Ethernet wire protocol specifies a redundant coding scheme that encodes successive eight bits of data with two bits of error correction data. This is known as 10/8 encoding. With 10/8 encoding, a 10 Megabit Ethernet card has a nominal data rate of 1 Megabyte/sec, a 100 Mb Ethernet NIC transmits data at a 10 MB rate, etc. Similarly, the 10 Gb Ethernet card has the capacity to transmit application data at 1 GB/sec.

To understand the rate interrupts need to be processed on a host computer system to sustain 1 GB/second throughput with Ethernet, simply divide by the average packet size. To keep the math easy, assume an average packet size is 1k or less bytes. (This is not an outlandish assumption. A large portion of the Receive packets processed at a typical web server are TCP ACKs; these are minimum 50 byte headers-only packets. Meanwhile, http Get Requests containing a URL, a cookie value, and other optional parameters can usually fit in a single Ethernet 1500-byte packet -- in practice, the cookie data that most web applications store is often less than 1 KB.) Assuming an average packet size of 1 KB, a 10 Gb Ethernet card that can transfer data at 1 GB/sec rate has the capability of generating 1 million operations/sec on your networking server. Next, assume there is a 1:1 ratio of Send:Receive packets. If 50% of those are Receive operations, then the machine needs to support 500K interrupts/sec.

Now, if the number of instructions associated with network interrupt processing in the Interrupt Service Routine (ISR) associated with the device, the Deferred Procedure Call (DPC), and the next higher layers in the Network Device Interface Specification (NDIS) stack to support TCP/IP is, let’s say 10,000, then the processor load to service TCP/IP networking requests is:

500,000 interrupts/sec * 10,000 instructions = 5,000,000,000 instructions/second (1)

which easily exceeds the capacity of a single CPU in the many-core era. If these network interrupts are confined to a single processor, which is the way things worked in days of yore, host processor speed is a bottleneck that will constrain the performance of a high speed NIC.

Of course, instead of wishing and hoping that TCP interrupt processing could be accomplished within 10K instructions in today’s complex networking environment, it might help to actually try and measure the CPU path length associated with this processing. To measure the impact of the current TCP/IP stack in Windows Vista, I installed the NTttcp Test tool available here and set up a simple test using the 1 Gb Ethernet NIC installed on a dual-core 2.2 GHz machine running Windows Vista SP1 over a dedicated Gigabit Ethernet network segment. Since the goal of the test was not to maximize network throughput, I specified 512-byte sized packets and was careful to confine the TCP interrupt processing to CPU 0 using the following NTttcp parameters:

ntttcpr -m 1,0,192.168.3.51 -a 16 -l 512 -mb -fr -t 120

I was also careful to shut down all other networking applications on my machine for the duration of the test.

Here’s the output from a 120 second NTttcp run, allowing for both a warm-up and cool down period wrapped around the main test:

Throughput(KB/s)	16,475.553
Throughput(Mbit/s)	131.804
Average Frame Size	764.394
Packets Sent	1,309,892
Packets Received	2,586,923
Packets received/Int)	2
Interrupts/sec	9,494.04
Cycles/Byte	129.3

On the dual-core machine, CPU 0 was maxed out at 100% for the duration of the test – evidently, that was the capacity of the machine to Receive TCP/IP packets and process and return the necessary Acknowledgement packets to the Sender. I will drill into the CPU usage statistics in a moment. For now, let’s focus on the interrupt rate, which was about 9500 interrupts/sec or slightly more than 100 μsecs of processing time for each Interrupt processed. This being a 2.2 GHz machine, 100 μsecs of processing time translates into 220,000 cycles of execution time per TCP/IP interrupt. Substituting this more realistic estimate of the CPU path length into equation 1 yields

500,000 interrupts/sec * 200,000 clocks = 100,000,000,000 instructions/second (2)

a requirement for 100 GHz of host processing power to perform the TCP/IP processing for a 10 Gb Ethernet card running at its full rated capacity.

Next, I re-executed the test while running the xperf ETW utility that is packaged with the Windows Performance Toolkit to capture CPU consumption by the TCP/IP stack:

xperf -on LATENCY -f tcpreceive1.etl –ClockType Cycle

According to the xperf documentation, the LATENCY flags request trace data that includes all CPU context switches, interrupts (Interrupt Service Routines or ISRs) and Deferred Procedure calls (DPCs). As explained in [1], Windows uses a two-step process to service device interrupts. Initially, the OS dispatches an ISR to service the specific device interrupt. During the ISR, further interrupts by the device are disabled. Ideally, the ISR performs the minimum amount of processing time possible to re-enable the device for interrupts and then schedules a DPC to finish the job. DPCs are dispatched at a lower priority than ISRs, but above all other functions in the machine. DPCs execute with the device re-enabled for interrupts so it is possible for the execution time of the DPC to be delayed because it is preempted by the need to service a higher priority interrupt from the NIC (or another device).

Gathering the xperf data while the NTttcp test was running lowered the network throughput only slightly – by less than 2%, additional measurement noise that can safely be ignored in this context. The kernel trace events requested are basically being gathered continuously by all the diagnostic infrastructure in Windows anyway. The xperf session merely gathers them from memory and writes them to disk. The disk was otherwise not being used for anything else during this test and there was an idle CPU available to handle the tracing chores. The overall performance impact of gathering the trace data was minimally disruptive in this situation.

I then loaded the trace data from the .etl file and used the xperview GUI application to analyze it. See Figure 2.

Figure 2. xperview display showing % CPU utilization, % DPC time, and % Interrupt time, calculated from the kernel trace event data recorded during the NTttcp test execution.

Figure 2 shows three views of the activity on CPU 0 where all the networking processing was performed. The top view shows overall processor utilization at close to 100% during the TCP test, with an overlay of a second line graph indicating the portion specifically associated with DPC processing, accounting for somewhere in excess of 60% busy. The DPC data is broken out and displayed separately in the middle graph, and the Interrupt CPU time is shown at the bottom (a little less than 4%).

xperfview allows you to display a Summary Table that breaks out Interrupt and DPC processor utilization at the driver level, sorted by the amount of processor time spent per module. For the DPCs, we see the following.

Module	Function	Count	Max Duration [ms]	Avg Duration [ms]	Duration [ms]
ndis.sys		425125	1.116595	0.075376	32044.44967
	0x8ac79237	423800	0.797987	0.075552	32019.18583
	0x8ad38209	207	1.116595	0.11752	24.326752
	0x8ad3892f	1117	0.012439	0.000837	0.935867
	0x8ad399b3	1	0.001213	0.001213	0.001213
USBPORT.SYS		8312	0.064506	0.011802	98.100399
tcpip.sys		4154	0.551394	0.009585	39.817004
dxgkrnl.sys	0x8f34e09b	3039	0.528346	0.012848	39.047187
iastor.sys		1221	0.033546	0.015061	18.390545

Table 1. xperfview Summary Table display showing processor utilization by DPC.

Confirming my back-of-the-envelope calculation presented earlier, xperf trace data indicates the average duration of an ndis.sys DPC used to process a network interrupt was 75 μsecs. The total amount of time spent in DPC processing was approximately 32 seconds of the full trace, which lasted about 52 seconds, corresponding to slightly more than 61% busy on CPU 0.

Module	Function	Count	Max Duration [ms]	Avg Duration [ms]	Duration [ms]
ntkrnlpa.exe	0x828d6fa2	423803	0.023875	0.003173	1345.147547
dxgkrnl.sys	0x8f3630ea	3040	0.096759	0.039523	120.151742
USBPORT.SYS	0x8f4098c2	5529	0.025199	0.007241	40.038229
pcmcia.sys	0x82f8deea	4968	0.02401	0.006754	33.558272
iastor.sys	0x8aaa7f6c	4968	0.016345	0.005103	25.353482

Table 2. xperfview Summary Table display showing processor utilization by ISR.

The Summary Table display reproduced in Table 2 serves to confirm a direct relationship between the ndis DPC processing and the kernel mode interrupts processed by the ntkrnlpa ISR. The average duration of an ntkrnlpa ISR execution was just 3 μsecs. Together, the ISR+DPC time was just under 80 μsecs. This leads to a slight downward revision of equation 2:

500,000 interrupts/sec * 175,000 clocks = 88,000,000,000 instructions/second (3)

which remains a formidable constraint, considering the speed of current processors.

Finally, I drilled in the processor utilization by process, which showed utilization by the NTttcp process, whose main processing thread was also affinitized to CPU 0, at the receiving end of the interrupt responsible for an additional 11% CPU busy. Allowing for OS scheduling and other overhead factors, these three workloads account for the 100% utilization of CPU 0.

Process	Cpu Usage (ms)	% Cpu Usage
Idle (0)	32730.11165	50.4
NTttcpr.exe (3280)	7120.518872	10.97
services.exe (700)	582.050622	0.9
InoRT.exe (2076)	368.519663	0.57
dwm.exe (4428)	300.59808	0.46
System (4)	256.802505	0.4
svchost.exe (1100)	166.479679	0.26
taskmgr.exe (7092)	162.960941	0.25
msiexec.exe (6412)	145.122854	0.22
sidebar.exe (5884)	111.86257	0.17
WmiPrvSE.exe (7952)	93.797334	0.14
csrss.exe (668)	79.331035	0.12
svchost.exe (1852)	70.65799	0.11

Table 3. xperfview Summary Table display showing processor utilization by process.

Note that the NTttcp program responsible for processing the packets it receives probably represents a network application that performs the minimum amount of application-specific processing per packet that you can expect. It ignores the data payload contents completely, and its other processing of the packet is pretty much confined to maintaining its throughput statistics. We should also note that is a user mode application, which means that processing the receive packet does require a transition from kernel to user mode. It is possible to implement a kernel mode networking application in Windows – the http.sys kernel model driver that IIS uses is one – that avoids these expensive processor execution state transitions, but they are the exception, not the rule. (And, when it comes to building HTTP Response messages dynamically using ASP.NET, even http.sys hands off the HTTP Request packet to an ASP.NET user mode thread for processing.)

The point of this set of measurements and calculations is not characterize the network traffic in and out of a typical web server, but to understand the motivation for recent architectural changes in the networking stack – both hardware and software – to allow network interrupts to be processed concurrently on multiple processors. Those architectural changes are the subject of Part II of this blog.

Part II of this article is posted here.

[1] Some have argued otherwise. See D. D. Clark, V. Jacobson, J. Romkey and H. Salwen, “An analysis of TCP processing overhead.” IEEE Communications Magazine, 27(6):23-29, June 1989. This assessment was made prior to the overhaul of the TCP protocol proposed by Van Jacobson that was implemented to address serious scalability issues that Internet technology faced in the early years of its adoption. Taking account of both security and performance considerations, the TCP/IP protocol software stack as implemented today is considerably more complex. Microsoft Windows Server 2003 TCP/IP Protocols and Services Technical Reference by Davies and Lee is a useful guide to the full set of TCP/IP services that are provided today, except it does not include the additional functions in the Microsoft Windows Server 2003 Scalable Networking Pack release discussed here. For a recent description of TCP/IP host processor overhead, see Hyun-Wook Jin, Chuck Yoo, “Impact of protocol overheads on network throughput over high-speed interconnects: measurement, analysis, and improvement.” The Journal of Supercomputing 41(1): 17-40 (2007

Introduction to Control Theory and Its Application to Computing Systems -- Part 1

MarkBFriedman — Tue, 03 Jun 2008 01:37:00 GMT

Part 1 of the slide presentation.

Introduction to Control Theory and Its Application to Computing Systems -- Part 2

MarkBFriedman — Tue, 03 Jun 2008 01:36:00 GMT

Part 2 of the slide presentation.

Introduction to Control Theory and Its Application to Computing Systems

MarkBFriedman — Tue, 03 Jun 2008 01:16:00 GMT

Our DDPE colleague Joe Hellerstein, along with Tarek Abdelzaher (University of Illinois), Yixin Diao (IBM), Chenyang Lu (Washington University), and Xiaoyun Zhu (HP), is presenting a seminar this week at ACM Sigmetrics.

The session Abstract follows:

Abstract.

Feedback control is central to managing computing systems and networks. For example, feedback is employed to achieve response time objectives by adjusting scheduling priorities and bandwidth allocations. Control theory provides a way to determine if feedback loops are stable (e.g., avoid wild oscillations), accurate in their control (e.g., achieve the desired response time objectives), and settle quickly to their steady state values (e.g., to adjust to workload dynamics). Recently, control theory has been used in the design of many aspects of computing, with a few examples of commercial products designed using control theory. Examples of where control theory has been used include: networking protocols (e.g., new versions of TCP/IP), real time systems, web servers, database servers, multi-tier computing systems, and workload managers.

This tutorial provides an introduction to control theory for computer scientists with an emphasis on applications. The topics covered include: (1) Introduction to control theory concepts and discrete-time linear systems; (2) Application to self-tuning memory management in IBM’s DB2 Universal Database Management System; (3) Application to model-predictive control in distributed real time systems; (4) Application to automated workload management in virtualized data centers; (5) Application to managing power and performance in data centers; and (6) Research challenges.

The Powerpoint slides that accompany the presentation are attached.

Visual Studio Performance Testing -- Noise is Enemy #1

MarkBFriedman — Tue, 20 May 2008 03:25:00 GMT

Performance testing is essential to our quest to make Visual Studio provide a highly responsive user experience.

We do performance testing early and often. Before a new feature is checked into the main branch, a test build is created, and 100 to 200 tests are run to assess performance. These tests include scenarios for start-up, debugging, project build, editor interactions, and much more.

This sounds like a lot of work. Well, it is. But there’s even more work that takes place before a single bit is tested.

Here’s a brief summary of what we do. First, we work with Visual Studio product units (e.g., C#, debugger, XML editor, and about 30 others) to identify performance-sensitive user scenarios and to set response time goals for these scenarios. The product teams develop test cases for their scenarios. Our team, Developer Division Performance Engineering, reviews these tests to see if they meet certain acceptance criteria. Once a test has been accepted, it is incorporated into the division-wide suite that is used to assess if performance has degraded on a check-in. We call this a performance regression.

Of course, we need a test infrastructure as well. We have a hundred machines dedicated to performance testing. We also have automation software that runs tests, analyzes the results, and notifies product teams of performance regressions. Initially, our perspective was that automating performance testing has unique requirements that demand a unique infrastructure. Now, we recognize that performance testing has much in common with functional testing. Further, we want to use features such as load testing and test case management that are part of Visual Studio Team Test, a direction that is being pursued by Developer Division as a whole (in part to “dog food” Visual Studio features).

During a performance test, we collect a large number of PerfMon counters, CPU profiles, ETW traces, and more. However, the decision whether there is a regression (and hence if the changes can be checked-in) currently depends on just two metrics – CPU Time and Elapsed Time. If either of these is “significantly” higher than they are supposed to be, the check-in is rejected. (We deal with memory and I/O in other ways.)

But what constitutes significantly higher CPU Time and Elapsed Time? This question has been a source of animated discussions with developers eager to check-in their changes.

In the past, we used an adaptive but somewhat opaque approach to establish regression thresholds. The adaptive part allowed us to automatically reset performance baselines for regression thresholds, but it was done in a complex way. As a result, there were many times when developers contested that there was a performance regression because there was confusion about the performance baseline. One of the lessons we learned is that effective performance testing requires a transparent methodology for setting regression thresholds.

Our latest approach is to periodically set regression thresholds based on the results of multiple runs of the same baseline build. This provides a clear basis for comparing performance results on check-in.

This approach has another advantage as well – it allows us to eliminate certain sources of noise. For example, variations in the chip sets used on our test machines caused some machines to have very long I/O times (in excess of 20% longer). As a result, we developed a machine acceptance procedure whereby we eliminate these anomalous machines. There are concerns about tests as well. For example, some tests use very little CPU and so a seemingly significant increase in CPU Time can be due to very small changes in the way context switching takes place during the run. Indeed, even for CPU bound work, we have noticed CPU Time can vary by 5% to 10% due to changes in the way context switching occurs.

These are all examples of test noise. Properly addressing test noise is critical to the effectiveness of the performance engineering process. Nothing is more frustrating to a developer than to spend several days pursuing a performance bug that turns out to be noise.

Test noise is any characteristic of a test that makes it so that CPU Time and/or Elapsed Time vary so much that we can’t reliably determine if performance has improved or degraded. To assess whether a test is too noisy, we run it multiple times on the same machine with a baseline build, and see if we get similar results. By similar, we mean that results do not vary more than 10% from the mean of the run.

Below are results from a test with little noise for CPU Time. The horizontal axis shows the run, and the vertical axis is CPU Time in milliseconds. The asterisks are test executions or iterations that are done repeatedly within a run. In most cases, we do not include the first iteration because it often differs a great deal from the others due to initialization effects. (Of course, for start-up scenarios, we only use the first iteration!)

We look for several things to see if a test is too noisy. As in the figure above, we want all of the performance counters to lie within the range of plus or minus 10% of the mean. Also, we look for a fairly even scatter of performance counters below and above the mean, but with no discernable pattern. If there is a pattern, it suggests that something is systematically biasing the test and needs to be fixed.

Now, let’s look at some noisy tests. One case is that the test may actually be addressing more than one scenario with very different performance characteristics. Here’s an example of a test that appears to have this multiple scenario character:

Another possibility is that the test may occasionally do something that takes a lot longer, or the test may be so short that it is subject to intermittent activities occurring in the OS (e.g., lazy writes) or the CLR (e.g., garbage collection). Here’s an example of a test result that appears to have this kind of problem:

In this case, there are two extreme values. At first glance, it may seem that there is another problem as well since several of the iterations have values below the “mean-10%” line. However, this is an artifact of having two large values that make the mean larger.

It’s hard work to do all of this. But performance testing was central to performance gains in Visual Studio 2008 compared with previous releases. And, with the help of changes in our handling of test noise, more performance improvements are on the way.

We will share the best practices for performance testing that we develop for Visual Studio (and other Developer Division products) with our customers. Longer term, we hope to incorporate these best practices into Visual Studio tools so that you can be more effective with your performance testing.

by Joe Hellerstein, Ph.D.

Senior Architect, Developer Division Performance Engineering

Joe Hellerstein is the author of Feedback Control of Computer Systems, published by Wiley-IEEE Press in 2004. He had a long and distinguished career at IBM Research prior to joining Microsoft in 2006.

Thoughts on Intel's recent hardware announcements

MarkBFriedman — Tue, 01 Apr 2008 03:56:00 GMT

Intel briefed customers recently about the evolution of its processor architectures to support ManyCore processors. Highlights of the press briefing include announcing the quad-core Tukwila processor that supports the IA-64 Itanium architecture and a six-core x64-based processor called Dunnington that will be available later this year. The major focus of the announcement though was the new Nehalem architecture processors which are scheduled for production by the end of this year.

Intel is executing on an aggressive, two-year product cycle, driven primarily by semiconductor fabrication improvements in the first year of the cycle that double the amount of circuitry they can fit on a chip. These are followed up by architectural improvements designed to leverage the new chip density in the year following.

Last year, this involved bringing online a new 45-nanometer (nm) fabrication plant that utilizes a new technology, namely, hafnium-based high-k gate dielectrics and metal gates. This technology is designed to address a serious problem with the previous materials used to insulate the tiny circuitry etched into the substrate. Historically, the material used for this insulation has been silicon dioxide. As the dimensions of the circuit have shrunk with each succeeding generation of semiconductor fabrication technology, so has the insulation layer. At the 90 nm point, the silicon dioxide insulator was the width of just 5 atoms. Material scientists felt it could not be shrunk any further without ceasing to function as an effective insulator. At that point, it was also subject to leaking power significantly (and generating excess heat). It started to look like we had reached a physical limit on how much circuitry could be crammed onto a single chip.

Intel is championing the new materials and manufacturing processes as a breakthrough that will enable it to build increasingly denser chips. (If high-k gate dielectrics really is a breakthrough, the rest of the semiconductor industry can be expected to follow.) The Tukwila and Dunnington processors announced last week each contain close to 2 billion logic circuits. Intel expects that the another two-year product cycle will hit like clockwork next year when it moves to a next generation 32-nm fab. This will once again double the number of circuits that can be packed on a chip to 4 billion. Enter the new processor architecture, code-named Nehalem designed to exploit the new fabrication density.

The Nehalem architecture is a modular design that can be produced today on the 45 nm process, but, can migrates next year to the 32 nm process. This year it will be built with 4 processor cores, next year 8. Consistent with the imperative to conserve power, it does not look like the Nehalem will increase the CPU clock speed. At least there was nothing about a faster clock in the announcement materials.

The emphasis on conserving power is leading to an increased interest in utilizing Simultaneous Multi-threading (SMT) technology to boost processor performance. This is something Intel branded initially as Hyper-Threading (HT). It is nice to see Intel moving away from the original, marketing-oriented branding, which was confusing, to the generic and generally accepted terminology. The announcement hints at expanding the use of SMT in future Nehalem chips, which is something the original SMT research found to be very promising. We could get 8 processor cores on the 32 nm fab process next year, each supporting up to 4 logical processors on a desktop machine. I don’t know if that will happen because it is not clear that desktop machines need anything close to 32 processors. That’s the challenge that software development has to step up to because right now, as I blogged about last time, the current generation of desktop software cannot effectively utilize all those processors.

One of the performance issues that arose with HT in earlier Intel processors, especially with server workloads, was that the added logical processors had a tendency to saturate the Front-side bus (FSB) that connected the processors to the memory controller. (Here’s an anecdotal example: http://www.cmg.org/measureit/issues/mit15/m_15_2.html.) The bus accesses are needed so that the snooping protocol can maintain cache coherence in a multiprocessor. The new QuickPath Interconnect provides a scalable alternative to the FSB that has both better latency and higher bandwidth. This should help bring SMT into the mainstream for server workloads. Architecturally, QuickPath looks very similar to AMD’s HyperTransport (another “HT,” which may explain why Intel has reverted to using the generic SMT terminology).

In the QuickPath architecture, each microprocessor contains a built-in memory controller designed to access its own dedicated local memory. But on machines configured with more than one microprocessor, QuickPath leads to a NUMA architecture. Within a microprocessor, all the processor cores and their logical processors can access local memory at a uniform speed using the integrated memory controller. But access to remote memory on a different microprocessor is slower. A program thread running on one microprocessor can access remote memory attached to another microprocessor using the QuickPath Interconnect.

These are the basic performance characteristics associated with NUMA (non-uniform memory access) machines. NUMA used to be associated with mainly esoteric high-end super-computers, mainly due to the difficulty developers had in programming for them. Even so, NUMA is poised to become a mainstream architecture, which is another serious challenge for our software frameworks.

To run well, a multi-threaded program running on a NUMA machine needs to be aware of the machine environment and understand which memory references are local to the node and which are remote. A thread that was running on one NUMA node that migrates to another node pays a heavy price every time it has to fetch results from remote memory locations. The Windows OS is already NUMA-aware to a degree. Once dispatched, threads have node affinity, for example. And Windows OS memory management is also NUMA-aware. The OS resists migrating threads to another node and otherwise also tries to ensure that most memory accesses remain local using per node memory management data structures. There are a number of NUMA-oriented APIs that applications can use to keep their threads from migrating off-node and direct memory allocations to a specific physical processing node.

In the meantime, it is an open question how NUMA-aware the application level run-times and critical memory management functions like the GC in .NET need to be. Let’s assume for a moment that 8 processor cores is more than enough for almost any desktop. This means dealing with the complexities of NUMA is confined, in the short term at least, to server applications. While that is occasion for a big sigh of relief, it is also a pointed reminder that facilities like the new Task Parallel Library in the .NET Framework will need to become NUMA-aware.

Finally, it is worth mentioning that there are a bunch of other goodies in the recent Intel announcement to further fuel the drive to many-core processors . These include making the serializing instructions like XCHG run faster, which should be a boost to multi-threaded programs of all stripes. Intel is also adding a shared L3 cache to each chip; each processor core continues to have its own dedicated L1 and L2 caches.

Where Do We Go From Here, Part 1.

MarkBFriedman — Sat, 22 Mar 2008 03:53:00 GMT

The Performance of Desktop Applications in the ManyCore Era

The Quad-cores are coming! The Quad-cores are coming!

Beginning in early 2008, machines with the latest quad-core processors became available from the major manufacturers. Should you be excited about the prospect? Should you run right out and buy one?

These machines will have 4 independent processor cores on a single processor chip, each able to run independent OS or application program threads concurrently. The processor core terminology is designed to distinguish this architecture from processors that support simultaneous multithreading (SMT) technology like Intel’s HyperThreading (or HT) that simulates multiprocessing on a single processor core. Quad-core means four physical processors reside on the chip; eventually, with HT-enabled you can then simulate 8 logical processors on a single chip (supporting 8-way concurrency).

These latest multi-core machines represent an extension of an ongoing trend in processor design that has become known as the ManyCore architecture. Instead of machines with significantly faster CPUs, with ManyCore we expect machines with more physical processors on the chip. And these new parallel processing chips are directed at standard server, desktop, and portable computers. This year’s crop of quad-core machines are then expected to be replaced by a subsequent generation of ManyCore machines with 8, 16, 32, 64, or more processor cores over the next 5-10 years. These developments in computer hardware present a serious challenge for the software development community to harness this computing power effectively in the next generation of computer software.

In this series of blog entries, I am going to look at that challenge from several perspectives, including the advances in software development technology that are necessary to make many-core processors yield their full potential. Multiprocessor scalability and the software necessary to exploit massively parallel processors is such a wide and deep topic in computer engineering that it is difficult to tackle head-on. In this first installment, I will focus on the factors influencing the hardware evolution. It also discusses why many-core processors are happening now. And, finally, I will talk about what to expect in the way of scalability & performance from this hardware running both current application software and the software of the future that will be designed to exploit the Manycore platform fully.

Along with the move towards multiple CPUs on a single multiprocessor chip, the discussion will highlight several related hardware developments, including power management, improved timing features, 64-bit computing trends, non-uniform memory access speeds, new instructions sets, and even a word or two about the impact of virtualization technology.

First, let’s look at the performance and scalability implications of deploying multiprocessors to run current Windows desktop and server applications. This will provide a broader perspective on some of the key performance-oriented challenges associated with many-core processors.

To illustrate these challenges, I took a screen shot from a performance monitoring session I ran showing utilization on my desktop computer for one of my somewhat typical days as a Knowledge Worker. During the course of this particular day, I attended a couple of meetings when my machine was virtually idle. (Ah, meetings, I just love them!) But while I was working at my desk, I was mostly using my computer. I answered a bunch of e-mails, surfed the web a few times, and even managed to find some time to do some programming and software testing. Processor utilization across a dual-core two processor machine at one minute intervals throughout the day averaged less than 10%. Even when the machine was busy, it wasn't very. Processor utilization peaked at less than 40% busy.[1] Given this feeble load on my machine, explain to me again why I need more processing power. If the two processors I have currently are usually idle, wouldn’t having four of them mean just that much more idle horsepower?

This is just a bit of the strong evidence that suggests that the need for many-core processors on the average person’s desktop machine is currently quite limited. Therein lies the first and foremost challenge to the industry’s goal of replacing your current desktop hardware with a new machine with 4, 8 or more processors. The overall performance gains that this hardware promises absolutely require application software that can fully exploit parallel processing hardware. That software is generally not available today for typical desktop applications which were written with conventional single CPU machines in mind. Not enough of today’s software was designed with parallel processing in mind. But that is beginning to change as the software industry rushes to catch up with the hardware. (Nothing new about this, by the way – software technology usually lags the hardware by 3-5 years or more.)

But the situation is far from hopeless. Developers of desktop applications have not had good reasons to program for multiprocessors until recently. Consider, for example, the xBox 360 platform, which is a three-way 64-bit multiprocessor with a parallel vector graphics co-processor (GPU) that is capable of performing something like 20 billion vector operations per second. The rich, immersive user experience associated with a well-designed xBox 360 application is a harbinger of the future of the Windows desktop as we start to build desktop applications that fully exploit multi-core architecture machines.

But getting there will be a serious challenge. I say that for several main reasons:

· Although the software industry has made significant progress in parallel programming, there is no general purpose method available that can reliably take a serial process and turn it into a multi-threaded parallel processing application. While some parallel programming patterns like “divide-and-conquer” can be used in many situations, most applications still require parallelization on a case by case basis.

· Multiple threads executing in parallel have non-determinative execution patterns that introduce subtle errors that can be very difficult to debug with existing developer tools.

· If adding a little parallelism to a serial application often leads to major speed-ups, there are often diminishing returns from adding more and more parallel processing threads. Determining an optimal number of parallel processing threads can be quite difficult in practice.

More about the multi-core challenge and related topics in software performance in a future blog post.

[1] During lunch and the other periods when I wasn’t even at my desk, utilization of the machine never dropped below 2%, for which I can thank all those background threads in both Windows Vista and Office 2007 that wake up and execute from time to time, regardless of whether I am using the machine or not.

Who Am I and What Am I Doing Writing a Blog?

MarkBFriedman — Fri, 21 Mar 2008 04:58:00 GMT

My name is Mark Friedman and I have been working here at Microsoft as an Architect in the Developer Division Performance Engineering team since October 2006. Although I am a newbie here, I am an industry veteran with an extensive background in software product development, particularly in developing monitoring tools for finding and resolving performance problems. These experiences include designing and developing several major products that brought me a modest measure of fame and fortune.

Although I consider myself primarily a software developer, over the years, I have also written extensively on performance-oriented topics, including two books on the subject of Windows performance, most recently, the Performance Guide that is packaged with the Windows Server 2003 Resource Kit. You can find bunches of articles I have written over the years for the Computer Measurement Group annual conference and its MeasureIT online publication or at the web site of my previous company.

If you bother to track down some of the links above, you will notice that I have not been writing much for public consumption lately. This prolonged period of what might look like writer’s block is due to my starting on this job to orient myself around what is to me a new technical challenge, namely developing a deep understanding of application performance on the Windows platform. The Developer Division produces a number of major applications, including Visual Studio, the Visual Studio Team System, the Team Foundation Server, the Expression suite of designer tools for web development, the C++, C#, and Visual Basic compilers, the Common Language Runtime (CLR), and the .NET Framework. This is a very diverse set of software products, to say the least. When you factor the myriad ways customers can wield these developer tools and technologies to build their applications, the word “diverse” doesn’t seem adequate to the task. Something along the lines of Universal Turing Machine seems more apt.

Software Performance Engineering (SPE), which is my team’s charter, is the discipline associated with building responsive and highly scalable applications. The problem we face is how to apply the principles and best practices associated with SPE to the engineering practices Microsoft uses to build the complex software we sell to customers. Furthermore, we want to incorporate successful practices and procedures into the developer tools we sell to customers. This would help them develop Windows-based applications that meet stringent performance and scalability requirements.

One particularly sobering aspect of this challenge is that there is no fully realized blueprint anywhere in the software industry for successfully integrating the performance engineering discipline into the application development life cycle. This has been a problem nagging advocates of the SPE approach from its very inception. Today, for example, there are a number of excellent books available on the subject suitable for general practitioners, including Connie Smith and Lloyd Williams’ Performance Solutions: A Practical Guide to Creating Responsive, Scalable Software. Smith and Williams’ book is probably the most thorough practical guide available for the practitioner. I also happen to be particularly partial to Daniel Menasce, et. al., Performance by Design: Computer Capacity Planning By Example, but that approach is more informed by analytic modeling, rather than the software development lifecycle. But neither book is specific to the Microsoft platform in general and the .NET Framework in particular.

The performance engineering guide that comes closest to addressing the specific needs of the .NET developer is Improving .NET Application Performance and Scalability. This valuable book provides an enormous amount of practical guidance to the .NET developer. I don’t like to think about the Herculean effort it took to produce this extensive developer’s guide, mainly because it will probably be my responsibility to produce the next edition of it. It is almost encyclopedic in scope. Even so, in my view, the advice contained in the book could be improved in at least one key area, which would be the adoption of the empirical, measurement-oriented approach I advocated in my two Windows performance books.

The number of public APIs in the .NET Framework is enormous, not to mention its extensibility by third party component developers and your in-house programmers. These object-oriented components can then be assembled into an executable program in any imaginable combination. When you factor in the need to consider all possible (or likely) permutations of .NET classes and their methods, my belief is that no static set of developer guidelines can ever be adequate to the task. This, by the way, is not meant to diminish my enthusiasm for an initiative my colleague Rico Mariani calls performance signatures, which he has discussed extensively on his blog, which relies on static analysis. I am sure Rico is correct in the value of some form of static analysis. But, in addition to a static analysis, we also have to measure the application’s performance empirically to see if its execution is taking too long and then determine why. In my mind, the pattern developed in the relational database world around having both a static definition of the execution Plan chosen by the Optimizer and a dynamic Explain function is probably worth copying.

If you are a .NET developer interested in building a highly scalable application, you are currently pretty much on your own in trying to figure out how to get started. Fortunately, basic performance considerations for .NET developers are not that different than for any other computing platform. But the .NET Framework is so wide and deep, getting going is a daunting prospect. Picking up a copy of Improving .NET Application Performance and Scalability, which runs to slightly more than 1000 pages, is also quite intimidating.

This is also a problem I face in starting to write about this problem space. The subject is so broad and so deep, it is difficult to know quite where the best place to start is. Under these circumstances, blogging about the area more informally than I might otherwise attempt in writing about this subject seems like a reasonable way to get started. I also work collaboratively with a remarkably talented group of individuals here in the Developer Division who share my interest and passion for application performance. They are accomplished authors on their own, too. I will attempt to structure this blog as a team effort so you have a chance to read their contributions to our understanding of this area as well.

So, if you don’t mind reading a work in progress, please stay tuned. If following me down a blind alley from time to time is a prospect that will dishearten you, I suggest you wait for the polished book version. But I warn, you that is probably at least several years in the future. What I intend to blog about over the next few months is our first tentative steps to formulate an empirical approach to engineering .NET application scalability, informed both by my background in computer performance and access to the internals of Windows, the CLR, and the .NET Framework. Someday, this might form the makings of a decent book on the subject, perhaps version 2 of Improving .NET Application Performance and Scalability. But, fair warning, much of what we will be writing about is preliminary, as we work out the most effective set of techniques.

Initially, I expect you will see me writing on an unholy mix of three topics that I am currently thinking about the most:

1. Parallel computing, including the multi-core hardware that we can expect and the challenge being issued to programmers to exploit it in the software we develop. I am not sure that I have anything terribly original to say about this topic, but I would at least like to acknowledge its scope and report on what tangible progress I see being made here in this important area.

2. The performance of the Collection classes, which is really a proxy for a wider larger discussion about what software architects and developers need in terms of run-time instrumentation to deal effectively with performance and scalability considerations with the .NET Framework. There are currently some 30 million lines of code in the Framework, so it meets almost anyone’s definition of a huge surface area that developers must understand and interact with in order to be successful. Collections of one sort or another are a good choice to help frame this discussion because they are key elements of almost every substantial application that is built using .NET. Moreover, the choice of which Collection class to use in your application is one of the key decisions developers have to make. Some form of performance intellisense (RicoM’s phrase) to help catch poor choices during application design and coding would undoubtedly be a cool feature in Visual Studio. But, realistically, there is no way to say whether the developer has selected the right Collection class until we understand both the size (or cardinality) of the collection and the way it is accessed.

3. Finally, I plan to write a few blog entries on the subject of performance and scalability testing. Performance testing is another area where the Microsoft Patterns and Practices group has published a healthy amount of guidance. Hopefully, I can contribute something in this space that is specific to the .NET application platform.

<:AtomicElement><:HighlightText>

Photo credit: The copyright-protected artwork used in the logo is provided courtesy of James Neff. It is used with the permission of the artist. It is entitled “1947 Mt. Rainier, Washington.” The picture commemorates the detailed, widely reported sighting of UFOs by Kenneth Arnold, a businessman and pilot, in 1947 near Mt. Rainier in Washington, which led to the first coining of the term “flying saucers” to describe the encounter. Reproductions of the artwork suitable for framing are available for sale at http://www.rense.com/prints.htm. By the way, Mr. Arnold never claimed he saw saucer-shaped UFOs, but that was the terminology that captured the imagination of the public. He claimed he saw wing-shaped vehicles flying at great speeds. What he did marvel at was the way these UFOs seemed to float and move like saucers being skipped across a lake or stream.