Summer of Benchmarks: .NET vs. IBM WebSphere 7 Performance and Price Debate and Discussion

gregleak@microsoft.com — Mon, 17 Aug 2009 19:54:00 GMT

This is turning out to be the summer of benchmarks. Check out http://www.wholoveswindows.com/websphere for the latest update to the benchmark discussion, including the latest IBM and Microsoft responses. I spent the late spring finalizing and evangelizing a study, which I’ve previously discussed on my blog, that compares performance and price of Windows Server 2008/.NET Framework 3.5 systems to IBM Power6/WebSphere 7 systems. I scrutinized the results, took tips from the community and my colleagues to ensure accuracy of the tests and answered a lot of questions on the methodology. The bottom line is, I stand behind the testing methodology and the results, and I encourage customers to read the study and conduct the tests themselves. I’d also love to continue to get comments and feedback on your experiences, should you decide to take the plunge.

Showing customers how to save money and time with technology is something I care about a great deal. I also want to make sure customers can get equally solid real-world results, using the sample benchmark applications with published source code as learning guides. Full disclosure of such benchmarking and testing is extremely important, as it allows customers and competing vendors to fully analyze the results, and even replicate the testing on their own such that fully informed choices can be made. Without this level of disclosure, customer should question the credibility of their results, especially considering our specific technical counter-points included in this paper.

Turns out, I’m not quite done testing. IBM recently shared their own benchmarking results in an effort to challenge my original benchmarking results published in April 2009. Once I obtained a copy of IBM report detailing their results, I was excited to dive in and explore the findings. You can find my response to IBM’s challenge, as well as my original benchmarking results at our new website, http://www.wholoveswindows.com/websphere. The bottom line, I stand my previously published results and continue to invite IBM to meet us in an independent lab to perform the same tests of the .NET StockTrader and WSTest benchmark workloads and pricing analysis of the middle tier application servers tested in our benchmark report. In addition, we invite the IBM competitive response team to our lab in Redmond, for discussion and testing in their presence and under their review.

So, what’s next for my summer of benchmarks? It’s hard to say. I can tell you that I *hope* it includes a joint Microsoft/IBM effort to have results on the respective platforms validated by a third party!

Latest WebSphere 7 and .NET Benchmark Results Stir Debate

gregleak@microsoft.com — Thu, 14 May 2009 00:48:00 GMT

Well, if you have not noticed there are new benchmark results for the Trade application running as a Java application in WebSphere 7 and running as a .NET application, posted at http://msdn.microsoft.com/stocktrader. Also, Steven Martin, the head of my division, posted a really good blog post on these results, released last week, at: http://blogs.msdn.com/stevemar/archive/2009/04/30/websphere-loves-windows-who-knew.aspx.

Also, take a look at http://www.websphereloveswindows.com.

The thing that is new here, and sparking the usual debate that comes along with almost all benchmarks I do (including the infamous PetShop/MiddleWare/TheServerSide.com benchmarks I did in the past) is that we ran a WebSphere 7-version of IBM's trade application on a high-end IBM Power6 server--specifically a Power 570/AIX server that costs $215,000.00 even without any WebSphere licenses. We wanted to document the following:

How does WebSphere 7 running on Windows Server 2008 and an HP BladeSystem (using moderate scale out vs. a RISC scale-up approach on Power6) compare in performance and total cost to the Power 570 server? Hint: The HP BladeSystem costs about $51,000.00
How does the .NET implementation running on the same HP BladeSystem compare to both WebSphere 7 on Windows and WebSphere 7 on the high-end Power6/AIX system (both in performance and middle tier app server hardware + software costs)?

Checkout http://www.websphereloveswindows.com for summary, links to full benchmark paper, etc. You can use the provided Capacity Planner tool to test other hardware configs, comparing for example, WebSphere on a mainframe to WebSphere on other platforms; or .NET on a Windows Server 2008 platform or even Linux platform of your choosing.

As usual, my comparison has sparked some debate. In a thread below I re-post the latest rebuttal from an anonymous source that thinks the comparison is biased, along with my responses to the points brought up. I work hard on the benchmarks I run, and encourage this type of feedback. That is what full disclosure for such benchmarks, with published code, detailed tuning documents, test scripts, etc. is all about. So comment away!

Inside The Lab : WCF Performance

Summer of Benchmarks: .NET vs. IBM WebSphere 7 Performance and Price Debate and Discussion

Latest WebSphere 7 and .NET Benchmark Results Stir Debate