Mainstream NUMA and the TCP/IP stack: Part I.

Parallel Scalability Isn’t Child’s Play

Developer Division Performance Engineering blog — Mon, 16 Mar 2009 23:44:51 GMT

In a recent blog entry , Dr. Neil Gunther, a colleague from the Computer Measurement Group (CMG), warned

New NUMA Support with Windows Server 2008 R2 and Windows 7

Regarding Windows Server 2008 R2 — Wed, 24 Dec 2008 21:53:03 GMT

The 64-bit versions of Windows 7 and Windows Server 2008 R2 support more than 64 Logical Processors (LP)

Mainstream NUMA and the TCP/IP stack: Part II: Programming ccNUMA machines

(Semi) Official Developer Division Performance Engineering blog — Mon, 28 Jul 2008 19:51:55 GMT

This is a continuation of Part I of this article posted here . In Part 1 of this article, we looked at

re: Mainstream NUMA and the TCP/IP stack: Part I.

gopal — Wed, 02 Jul 2008 11:12:22 GMT

I hope you noted that when you tested and saturated the CPU , IPsec was probably enabled and that took most of your CPU. Because otherwise, CPU utilization would have been much less.( less than 25% on a mainstream dual core machine).

Yes, your concern remain valid, stack should be able to take advantage of multi-cores present on the machine.

re: Mainstream NUMA and the TCP/IP stack: Part I.

Mark B Friedman — Mon, 23 Jun 2008 19:37:52 GMT

Manu,

Thanks for the correction & amplification on ethernet bit rates.

The 1000BaseX standard is mainly for fiber connections where the nominal data rate is 1 GB sec.

In my test I was running 1000BaseT over cat 6 wiring, in which case it looks like I would get the following (which looks more like 8/12 encoding):

"The data is transmitted over four copper pairs, eight bits at a time. First, eight bits of data are expanded into four 3-bit symbols through a non-trivial scrambling procedure based on a linear feedback shift register; this is similar to what is done in 100BASE-T2, but uses different parameters."

So, if I am reading this correctly, the nominal data rate is even less that what I reported in the test I ran. I will look into this a little more and correct the post accordingly. Thanks, again.

Hey, what are a few megabytes/sec here and there between friends?

re: Mainstream NUMA and the TCP/IP stack: Part I.

manu — Mon, 23 Jun 2008 13:15:14 GMT

Thanks for your clarifications. I just wanted to raise clock vs Instruction issues and as you rightly pointed out these things are much dependent on cpu architecture and increasingly becoming complicated to calculate.

For ethernet line rates, you may want to look at

http://en.wikipedia.org/wiki/Gigabit_Ethernet

excerpt :

--------

"The IEEE 802.3z standard includes 1000BASE-SX for transmission over multi-mode fiber, 1000BASE-LX for transmission over single-mode fiber, and the nearly obsolete 1000BASE-CX for transmission over balanced copper cabling. These standards use 8B/10B encoding, which inflates the line rate by 25%, from 1000 Mbit/s to 1250 Mbit/s to ensure a DC balanced signal. The symbols are then sent using NRZ.."

re: Mainstream NUMA and the TCP/IP stack: Part I.

Mark B Friedman — Sat, 21 Jun 2008 00:51:16 GMT

Manu,

Thanks for the comment.

On the nominal data rate of an Ethernet link using 10/8 encoding, I am going to hold my ground. But you are certainly correct when you ask me to account for the Ethernet preamble, postamble, and the header itself. Ethernet is actually not capable of delivering the full nominal data rate to the next higher level protocol. I deliberately neglected that detail here, but it is an important detail that people are prone to overlook.

On the instruction execution rate vs. clock cycle rate, your comment pushed me to make a correction I should have made earlier. If you double-check above, I changed "instructions" to "clocks" (or cycles) in equation 2 & 3 because that is clearly what I was measuring here.

I should not have been so free about mixing up instruction execution rate and clocks, so thank you for making me clarify that.

In this article I wanted to avoiding discussing the Intel microarchitecture. But your question invites me to deal with it a little bit. Please understand that in writing "instructions" instead of "clocks" in Equations 2 & 3, I was deliberately being conservative. My expectation for this workload is that the CPI (cycles per instruction) would actually be less than 1, although I admit I did not go to the trouble of measuring that in this case.

To explain...

You are correct when you say instructions require multiple clock cycles to complete. Instruction execution is pipelined, so even simple instructions require multiple clock cycles to execute. In the original 486 and Pentium chips, the pipeline depth was 5, I believe. Starting with the P6 microarchitecture, though, things got considerably more complicated. The goal of a pipelined architecture is to achieve a CPI of 1, which for a variety of reasons (where it is necessary to stall the pipeline) it is almost impossible to get there. (For really deep background on this, I recommend reading "Computer Architecture" by Hennessy and Patterson. If that looks intimidating, try their introductory textbook "Computer Organization and Design.")

The current microarchitectecture aims to do better than 1 CPI. The processor does not execute native x86 or x64 instruction directly. It translates native instructions into micro-ops and these can be executed out of order & in parallel. The microarchitecture does ensure that instructions are "retired" in their original sequence. (The most important performance metric is the rate that native x86 Instructions are "retired" per sec.) With Intel processors today, the pipeline depth is model-dependent (and more like 10-20 cycles per instruction), but they are all capable of retiring as many as 3 or 4 instructions per clock cycle.

For critical code paths like the TCP/IP stack in Windows, it is safe to assume that you are executing optimized code that will retire more than 1 instruction per clock. That is why I felt safe using "instructions" without further explanation in this context, but "clocks" or "cycles" is still better.

Thanks, again.

-- Mark

re: Mainstream NUMA and the TCP/IP stack: Part I.

manu — Fri, 20 Jun 2008 20:47:31 GMT

"With 10/8 encoding, a 10 Megabit Ethernet card has a nominal data rate of 1 Megabyte/sec, a 100 Mb Ethernet NIC transmits data at a 10 MB rate, etc. Similarly, the 10 Gb Ethernet card has the capacity to transmit application data at 1 GB/sec.

this is rather wrong assumtion. One 1 Gbps ethernet link generally have a line rate of 1.25 Gbps to give 1 Gbps( 1000/8 = 125 GB/sec). Though because of overhead of ethernet frame actual data throughput to higher layer will be less.

And secondly, your instruction calculation is less convincing. 2.2 GHz does not mean 2.2 Giga instruction, it is just that many clock ticks. One instruction can take more than one tick.