UPDATE 1/7/2013: I recently re-ran the scenario on the original hardware and could not reproduce the RSS issue below. I found that by setting 'Number of RSS Queues' to 4 for each of the Broadcom NICs, I could get an even distribution of load across all 16 server cores. There was likely an RSS configuration issue, but I can't confirm what that was.
Re-running on the original hardware after setting 'Number of RSS Queues' to 4 for each NIC:
In the previous post, I stated that when adding multiple NICs to a web server you should ensure that the network load is balanced across those NICs. In order to maximize CPU on your multi-processor server you will still need to ensure that those NICs support Receive Side Scaling (RSS), a technology that distributes network processing across multiple CPUs. In this post I will demonstrate how to verify that RSS is working.
Recall our webserver specs:
After load-balancing our server and clients, we now have the following wcat node configuration:
After collecting data on the server NICs, enabling processor counters for all processors and running my wcat scenario, I was able to see what happened to incoming network traffic. In the the table below, notice that only 8 of the 16 CPUs processed the Deferred Procedure Calls (DPCs). Meanwhile, the other 8 CPUs spent more time Idle. Something in my server configuration was preventing RSS from utilizing all 16 cores.
To compare, we borrowed some hardware from the IIS performance lab which had previously gone through this RSS exercise.
The new webserver has a total of 20 processors and 10 NICs:
The following is the new load-balanced wcat node configuration:
And the following was the compilation of NIC and counter data for a wcat run. RSS was configured properly on this node – notice how the DPC processing was evenly distributed across all processors (taking into account that RSS CPUs 8-9 and e-f had increased load).
To compile this data, I did the following:
When running a web server with multiple NICs, be sure to verify that RSS is working properly. If you have questions, check with the NIC vendor and do your own testing once the NICs are installed.