Inside The Lab

Summer of Benchmarks: .NET vs. IBM WebSphere 7 Performance and Price Debate and Discussion

gregleak@microsoft.com — Mon, 17 Aug 2009 19:54:00 GMT

This is turning out to be the summer of benchmarks. Check out http://www.wholoveswindows.com/websphere for the latest update to the benchmark discussion, including the latest IBM and Microsoft responses. I spent the late spring finalizing and evangelizing a study, which I’ve previously discussed on my blog, that compares performance and price of Windows Server 2008/.NET Framework 3.5 systems to IBM Power6/WebSphere 7 systems. I scrutinized the results, took tips from the community and my colleagues to ensure accuracy of the tests and answered a lot of questions on the methodology. The bottom line is, I stand behind the testing methodology and the results, and I encourage customers to read the study and conduct the tests themselves. I’d also love to continue to get comments and feedback on your experiences, should you decide to take the plunge.

Showing customers how to save money and time with technology is something I care about a great deal. I also want to make sure customers can get equally solid real-world results, using the sample benchmark applications with published source code as learning guides. Full disclosure of such benchmarking and testing is extremely important, as it allows customers and competing vendors to fully analyze the results, and even replicate the testing on their own such that fully informed choices can be made. Without this level of disclosure, customer should question the credibility of their results, especially considering our specific technical counter-points included in this paper.

Turns out, I’m not quite done testing. IBM recently shared their own benchmarking results in an effort to challenge my original benchmarking results published in April 2009. Once I obtained a copy of IBM report detailing their results, I was excited to dive in and explore the findings. You can find my response to IBM’s challenge, as well as my original benchmarking results at our new website, http://www.wholoveswindows.com/websphere. The bottom line, I stand my previously published results and continue to invite IBM to meet us in an independent lab to perform the same tests of the .NET StockTrader and WSTest benchmark workloads and pricing analysis of the middle tier application servers tested in our benchmark report. In addition, we invite the IBM competitive response team to our lab in Redmond, for discussion and testing in their presence and under their review.

So, what’s next for my summer of benchmarks? It’s hard to say. I can tell you that I *hope* it includes a joint Microsoft/IBM effort to have results on the respective platforms validated by a third party!

Latest WebSphere 7 and .NET Benchmark Results Stir Debate

gregleak@microsoft.com — Thu, 14 May 2009 00:48:00 GMT

Well, if you have not noticed there are new benchmark results for the Trade application running as a Java application in WebSphere 7 and running as a .NET application, posted at http://msdn.microsoft.com/stocktrader. Also, Steven Martin, the head of my division, posted a really good blog post on these results, released last week, at: http://blogs.msdn.com/stevemar/archive/2009/04/30/websphere-loves-windows-who-knew.aspx.

Also, take a look at http://www.websphereloveswindows.com.

The thing that is new here, and sparking the usual debate that comes along with almost all benchmarks I do (including the infamous PetShop/MiddleWare/TheServerSide.com benchmarks I did in the past) is that we ran a WebSphere 7-version of IBM's trade application on a high-end IBM Power6 server--specifically a Power 570/AIX server that costs $215,000.00 even without any WebSphere licenses. We wanted to document the following:

How does WebSphere 7 running on Windows Server 2008 and an HP BladeSystem (using moderate scale out vs. a RISC scale-up approach on Power6) compare in performance and total cost to the Power 570 server? Hint: The HP BladeSystem costs about $51,000.00
How does the .NET implementation running on the same HP BladeSystem compare to both WebSphere 7 on Windows and WebSphere 7 on the high-end Power6/AIX system (both in performance and middle tier app server hardware + software costs)?

Checkout http://www.websphereloveswindows.com for summary, links to full benchmark paper, etc. You can use the provided Capacity Planner tool to test other hardware configs, comparing for example, WebSphere on a mainframe to WebSphere on other platforms; or .NET on a Windows Server 2008 platform or even Linux platform of your choosing.

As usual, my comparison has sparked some debate. In a thread below I re-post the latest rebuttal from an anonymous source that thinks the comparison is biased, along with my responses to the points brought up. I work hard on the benchmarks I run, and encourage this type of feedback. That is what full disclosure for such benchmarks, with published code, detailed tuning documents, test scripts, etc. is all about. So comment away!

New Interesting Benchmarks Coming: .NET Capacity Planner Tool

gregleak@microsoft.com — Sun, 11 Jan 2009 05:59:00 GMT

Just thought I would letfolks know of some fairly comprehensive benchmarks that I have been working on, involving IIS 7; multi-core scaling (up to 16 cores); WCF. And of course some comparative Data to Java will part of this. It has been an interesting few months; working through several hundred benchmark runs; with very careful attention paid to tuning, etc. The results, based on two workloads (StockTrader and Web Services WSTest) are based on a tool anyone can use; that I have fixed up lately. This is the Capacity Planner tool; that allows anyone to setup multi-agent benchmarks to test either/both of these workloads. Right now I am focussed on vertical scalability of services across SMP CPUs; as well as serialization performance between .NET IIS-Hosted; .NET self-hosted; HTTP-XML vs. TCP-Binary Encoding for WCF; and comparing to at least one popular Java Enterprise stack (just updated).

The results will be published on MSDN, in a new benchmark/tuning site I plan to create; separate but related to the StockTrader site. The cool thing is that everything gets published (including the test tool and Java implementations) as part of the StockTrader 2.03 download (http://msdn.microsoft.com/stocktrader ); so anyone can perform the tests form themselves on whatever equipment /software stack they want; helps in tuning and in guaging relative performance of various software stacks, including of courese .NET 3.5 and WCF.

-Greg

Apache support for the .NET StockTrader

gregleak@microsoft.com — Mon, 10 Nov 2008 22:19:00 GMT

This week at ApacheCon 2008, WS02 announced that they will be building an open source version of the .NET StockTrader sample application under “Project Stonehenge.” Driven by WS02, “Project Stonehenge” is a new Apache incubation project. They will use this project to set up sample applications to demonstrate seamless interoperability with multiple underlying platform technologies by using currently defined W3C and OASIS standards.

I think this is really exciting news, as it gives developers best practices for interoperable applications that communicate via distributed protocols. The project will demonstrate interoperability between platforms and provide sample code upon using service-oriented architecture (SOA), which developers can use in their own work. It also demonstrates cross-platform deployments of SOA technologies, which demonstrates both the technical and business value of SOA.

WS02 has been working with us since TechEd US last June on demonstrating interoperability using the .NET StockTrader reference application. I was on stage with them as they showed their demo during Bob Muglia’s keynote, and it’s great to see them to continue their support for the .NET StockTrader. We will also continue to support their interoperability efforts and consider participating in the future, including implementations of sample applications like the .NET StockTrader app. This project could offer a forum for important interoperability efforts around WS* and SOA, which can benefit the industry and customers.

Can’t wait to see what else we see in interoperability!

Video Clip - Interop Panel at TechEd IT Pro

gregleak@microsoft.com — Tue, 29 Jul 2008 06:16:00 GMT

Just in case you missed it last month at TechEd IT Pro, here’s a video clip of the Interop panel that I participated in with our industry partner, WS02, and moderated by Burton analyst, Chris Haddad. The main discussion points are interop topics that have affected all of us like, interoperability through SOA, WS-* standards and how they facilitate interoperability and common problems with interoperability. Worth taking a peek and let me know what you think.

My Demo at TechEd IT Pro Last Week

gregleak@microsoft.com — Sat, 21 Jun 2008 00:32:00 GMT

Just in case you missed me last week at TechEd IT Pro, I did a demo during Bob Muglia's keynote, and below is a clip of it. Enjoy!

.NET StockTrader 2.0 demo at TechEd IT Pro 2008

.NET Customer wins RDN's Innovator Awards

gregleak@microsoft.com — Fri, 13 Jun 2008 01:58:00 GMT

Just saw this interesting article in Redmond Developer News, which focuses on .NET customer, Southwest Florida Water Management District, winning the RDN Innovator Awards. The Southwest Florida Water Management District gained acclaim, in the SOA category of RDN’s First Annual Innovator Awards in the SOA category, with a well-construction permitting application that ties into a host of legacy systems. The SOA implementation, built on the Microsoft .NET Framework, ties together disparate Cobol applications, Oracle databases, an enterprise content-management repository, and a geographic information system to process 86 percent of the nearly 17,000 permits processed electronically.

Really great to see customers use .NET to implement SOA and this such a good example of how to leverage .NET!

.NET StockTrader 2.0 Demo at TechEd IT Pro

gregleak@microsoft.com — Tue, 10 Jun 2008 20:00:00 GMT

What a crazy couple of weeks in Orlando! I was at TechEd Developer last week doing a couple of sessions on my app, .NET StockTrader 2.0, and still here this week to do a .NET StockTrader 2.0 demo during Bob Muglia’s keynote at TechEd IT Pro. I will also have our partner, WS02, joining us on stage during the demo to showcase the interoperability on an Apache platform. Part of Bob’s keynote address, the .NET StockTrader and Configuration Service 2.0 will show how a composite application, running across Windows-, IBM WebSphere- and Apache-distributed components, could dynamically virtualized against physical resources on-premises and across the cloud.

New interoperability features demonstrated with WS-* Advanced Web Service Security are below:

· Message level security interoperability with WSO2 open-source based Web Services Application Server (Java)

· Message level security interoperability with WSO2 open-source Web Services Framework for PHP

· Interoperability across three distinct platforms (Microsoft, PHP, Apache Axis2/C) via Advanced WS-* protocols and X.509 certificates on service clients and service hosts.

Also, this demo shows that interoperability is real and ready for customers to leverage today. .NET StockTrader and Configuration Services 2.0 let customers to take advantage of their existing services and applications while enabling them to easily scale out by automatically detecting when you’ve added new computing nodes to the network and take advantage of scaling out with greater power. Real-world IT environments are a mix of technologies and we continue to prioritize to help customers achieve interoperability between systems for the benefit of sharing information across organizational and corporate boundaries.

There’ll also be a TechEd Online Interop panel discussion on Tuesday, June 10 at 2:00-3:00 pm ET with WS02, Sun and TIBCO to discuss how interoperability is important today and the role it plays for customers in the future. The panel will be held in the TEO are by the Microsoft store so if you have free time then, you should come by and watch the panel. It’ll be moderated by Burton analyst, Chris Haddad, and will be discussing the importance of interoperability today in our industry. The panel will also be taped so if you can’t make it, you can also check it online on the Tech·Ed Online Web site.

.NET StockTrader 2.0 and Configuration Services 2.0 Download Available!

gregleak@microsoft.com — Tue, 03 Jun 2008 23:41:00 GMT

Just wanted to provide everyone an update that the .NET StockTrader 2.0 and Configurations Services 2.0 are both available for download on the MSDN .NET StockTrader site, http://msdn.microsoft.com/stocktrader!

Know a lot of people were waiting anxiously for this and now it’s finally ready for you to try out. Also, I really do want your feedback on this app so please provide some comments on the developer forum, and we may use this feedback for future .NET StockTrader updates.

Let me know what you think!

.NET StockTrader Sessions at TechEd Developer 2008

gregleak@microsoft.com — Mon, 02 Jun 2008 21:01:00 GMT

I'm doing a couple of sessions this week at TechEd Developer in Orlando and if you're already attending, you should check it out. Hope to see you there!

Date	Timeslot	Code	Title	Session Type	Speaker(s)
6/5/2008	10:15 - 11:30	SOA310	Interoperability Scenarios with Microsoft .NET and J2EE	BRK	Gregory Leake
6/6/2008	08:30 - 09:45	SOA306	Providing Load Balancing, Application-Level Failover, and Centralized Configuration Management with Windows Communication Foundation Services and Microsoft .NET Applications	BRK	Gregory Leake

More and more Interop…NET FX 3.5 SP1 Beta Bits Live!

gregleak@microsoft.com — Tue, 13 May 2008 18:16:00 GMT

In the last few weeks, we’ve seen Microsoft announce several Interop-related work that the company is doing, including BizTalk R3 with expanded interop with new Web Services registry capabilities with support for UDDI (Universal Description Discovery and Integration) version 3.0 and LOB applications, Microsoft BizTalk RFID Mobile - a light weight application that works on mobile RFID devices, and will enable delivery of relevant information from these devices to central business process systems, Interop news from partners, like SAP and TIBCO, and last, but not least the .NET StockTrader 2.0/Config Service 2.0. This also only just a few examples of Interop work that’s been going on at Microsoft.

Let’s face it. Interop is very important in the industry and Windows Communication Foundation makes it easier for developers to create seamless Web Service-based applications. Today, Microsoft released Service Pack 1 (SP1) betas for Visual Studio 2008 and the .NET Framework 3.5. This is very exciting for me because the SP1 for the .NET FX 3.5 delivers WCF improvements that give developers more control over the way they access data and services, including:

Interoperable object references
Enable WCF Service template in Visual Web Developer Express sku
Hosting Wizard for Visual Studio WCF Service projects in IIS
Enable Event Logging in Partial trust

These are only a few of the new features for WCF in the .NET FX 3.5 Beta bits. This is really important also for the investment of SOA applications which enables customers and developers to create more interoperable apps. As we see more companies involved in developing SOA apps like SAP and TIBCO as well as our own technology investments in developing more interoperable features in our products, we see the importance and need of SOA in the enterprise industry.

Also, go to Scott’s blog, ScottGu’s blog, to get more details.

What is Configuration Services 2.0?

gregleak@microsoft.com — Thu, 01 May 2008 22:36:00 GMT

Last week, I presented at different types of user group sessions including the .NET Connected Systems Group in NYC and the Scalability Group in Boston. I showed a new feature of my StockTrader app, Configuration Services 2.0, which makes it easier to build services easier with .NET Framework 3.5 and WCF, and you may have read more here, Microsoft Continues Its Work On Oslo -- SOA for Windows.

Here are a few of the main objectives of the Configuration Services 2.0 and this will all be posted in the white paper on MSDN on the .NET StockTrader site:

1. Make it easier to build services with .NET 3.5 and WCF

o Includes template bindings, security examples, interoperability examples with Java

o Manages host process, endpoints, connections between services automatically

o Designed for high-performance

2. Provide centralized configuration management of service nodes

o Move host and application-specific settings from config files to a repository approach

o Provide Web-based management UI

o Provide way to automatically update settings without host restarts

3. Provide these capabilities via easily re-usable shared libraries

o Easy for customers to implement in their own applications and services

o Provides base classes, uses .NET reflection

o ~20 lines of code and not much time investment for customer to implement!

You can also call this something like a distributed “Service Network” and is one of the first examples of a general purpose Service Network built using .NET managed code. These code libraries can be leveraged as part of your SOA projects to develop new composite application that mix/match services across both .NET and Java. The Service Network should be able to automatically detect when you’ve added new computing nodes to the network and take advantage of scaling out with greater power without requiring complex setup or configuration – it should auto-detect that more resources are available for use. You can read more about Service Network on Steven Martin’s blog.

Below is also a diagram that might make more sense. Let me know what you think and would love your thoughts on this.

Quick Update on my User Group Presentations

gregleak@microsoft.com — Mon, 28 Apr 2008 07:21:00 GMT

The last couple of weeks have been a whirlwind for me since I've been on the road presenting my .NET StockTrader 2.0 at user group meetings in San Francisco, New York City and Boston. Really want to provide more details on my experience on the road and will share more in the upcoming days, including the top questions I've been getting and my presentation that I've been showing. Just want to thank everyone for coming out to see me and giving me vaulable feedback on my new demo. It's been really beneficial for me and I'm greatful for you letting me come out. It just took shipping 5 servers to 3 different cities and setting-up a mini lab, but it was all worth it!

I'm off to London to showcase my .NET StockTrader 2.0 and Configuration Services 2.0 demo at the Microsoft Architect Summit, but keep your eyes out on MSDN since the .NET StockTrader 2.0 download will be up any day now, http://msdn.microsoft.com/stocktrader. Will also include a tutorial, new white papers and new Windows Server benchmarks.

Putting the Final Touches on .NET StockTrader2.0

gregleak@microsoft.com — Tue, 15 Apr 2008 20:45:00 GMT

Taped this video on Sunday when I was putting the final touches on my app and I talk a little bit about one of the new cool features, Configuration Services, and will definitely talk more about it during my presentations at the .NET User Group meetings this week and next. Also, look out for more blog postings on this too and want your feedback on it!

Final Touches of the .NET StockTrader 2.0

.NET StockTrader Benchmarks from Feb

gregleak@microsoft.com — Tue, 15 Apr 2008 20:29:00 GMT

As I finish up and put the final touches on .NET StockTrader 2.0, I wanted to share some benchmark findings that I did last February during the launch of the new versions of three cornerstone technologies: Windows Server 2008, Visual Studio 2008 and SQL Server 2008. Based on Windows Server 2008 and .NET Framework 3.5, I released new benchmarks for the .NET StockTrader, which also show that Windows Server and .NET can push perf boundaries against other publicly available apps like IBM’s J2EE StockTrader perf application.

Some of my favorite findings are:

· Microsoft Windows Server 2008 with .NET Framework 3.5 delivers 117% better throughput than IBM WebSphere 6.1 on Red Hat Linux for the Web Application Server test using the IBM-designed Trade 6.1 benchmark as well as delivers 93% better throughput for the remote services test.

· On the Sun Microsystems’ WSTest Web Services benchmark, .NET StockTrader demonstrates 94% better throughput on Windows Server on processing Web Service requests and 86% better throughput performance for the EchoStruct operation.

Check out the full white paper here, http://msdn.microsoft.com/stocktrader, which has a chart that better explains the high performance that I tested for the .NET StockTrader based on Windows Server 2008 and .NET Framework 3.5.

The paper on MSDN presents the benchmark results of two key application server workloads:

1. Trade 6.1 Application Server Benchmark created by IBM – This benchmark serves as IBM’s primary capacity planning tool for WebSphere, and as their primary performance sample application for Java Enterprise applications. The benchmarks detail throughput results for the IBM implementation vs. the functionally equivalent of the .NET Framework 3.5 implementation.

2. WSTest Web Services 1.5 Benchmark, created by Sun Microsystems – This benchmark tests an application server’s performance as a Web Service Host, measuring the platforms ability to process Web Service operations involving HTTP/SOAP requests, isolating the networking stack, Web server integration, and XML serialization engines within the application server.

The benchmarks also show that the .NET Framework is a proven foundation for enterprise mission critical applications and the misconceived perception that .NET isn’t scalable for enterprise apps. With Windows Server, the .NET Framework really does deliver impressive performance, scalability and interoperability relative to other options in the market. The .NET StockTrader is also a good proof point of interoperability as well since it has complete, seamless connections between front and back-end technologies. The sample app also illustrates that a developer who uses .NET can now focus on adding more features and functionality to your applications spending less time writing lines and lines of code and debugging.

I’m at my lab right now working on the new benchmarks for .NET StockTrader 2.0, which I hope to have posted on MSDN very, very soon so key an eye out for it!