WebSphere Loves Windows…Who Knew?

re: WebSphere Loves Windows…Who Knew?

Greg Leake — Thu, 14 May 2009 01:25:33 GMT

Interesting feedback and objections. Since I ran these tests over the past 6 months, I have re-posted your objections along with my responses on my blog at:

http://blogs.msdn.com/gregleak/

Greg Leake

Microsoft Corporation

PS: excellent debate!

Latest WebSphere 7 and .NET Benchmark Results Stir Debate

Inside The Lab — Thu, 14 May 2009 01:03:54 GMT

Well, if you have not noticed there are new benchmark results for the Trade application running as a

Rumor stops at wise

CK@SH — Wed, 13 May 2009 09:51:39 GMT

Though it is a waste of time to reply a "NO WARRANTIES, EXPRESS OR IMPLIED" document. Rumor stops at wise, I am willing to provide evidence for wise.

It is shameful to see such a highly respected senior personel at Microsoft standing behind such a flawed methodology and benchmark results, and still trying to claim victory. May I ask a few questions about the scenario to clear my mind?

Fairness comparison!

====================

1. IBM p570 vs HP c7000 is like using a truck vs 1000 bikes for mail delivery. 1000 bikes will be more efficient and cheaper, but try using 1000 bikes to deliver a nuclear reactor instead of a truck?

2. Please don't ignore p570's cost on Reliability Availability and Serviceability (RAS)! The idea of Price Performance Ratio comparison between P570 and HP c7000 misleads readers by ignoring the extra cost on Reliability Availability and Serviceability (or why you ordered AIX service?). It is obviously unfair and even ridiculous! If we find a middle school student who is better in java programming than your senior director, can I get a conclusion that this guy has high price performance ratio?Why order things that are not used in the paper, such as Advanced Power Virtualization, flat panel console, OS maintenance, 2 DVD-RAM drives, etc, while not making equivalent order in the blade system?

What is fair then?

==================

3. IBM p570 with AIX vs. HP Integrity with Windows 2008 DataCenter.

4. IBM BladeCenter H vs. HP c7000.

Flaws in cost comparison!

=========================

5. Why order things that are not used in the paper, such as Advanced Power Virtualization, flat panel console, OS maintenance, 2 DVD-RAM drives, etc, while not making equivalent order in the blade system?

6. HP hardware price: why no Ethernet switch in the chassis, no console kit, etc?

7. WebSphere AppServer price includes support, .NET solution has none.

Misleading in Test bed architecture:

====================================

8. Why using only 1 instance of WAS against 2 database servers? It's not possible to make use of both of them.

9. It would be fairer to run 4 instances of WAS on p570, distributing SQL workload to both database servers equally.

10. The p570 has 8 cores, 32GB memory and 4 NICs, while the c7000 has 16 cores, 128GB memory and 8 NICs combined. They are not equivalent hardware or architecture, but the wide difference tells a lot.

Misleading in Middleware optimization

=====================================

11. EJB is heavier for simple tasks, but better for high volume workload such as this one. Why is servlet mode being used for WebSphere Trade on p570?

12. Setting WebSphere on p570 with 32GB memory to run at max heap size of 2GB, why waste the remaining 30GB? You could run 14 of such instances concurrently!

13. Setting Trade to run at servlet mode with maximum web container threads at 50 on a 8-core p570? It means you're using 8 cores to run only 50 threads max, while this system was designed to accommodate thousands of concurrent threads.

14. On the p570 where only 1 WebSphere instance is run, setting maximum JDBC connection pool to 51 while each database server can accept 400 connections, what's the point of this limitation? In the .NET scenario there are 4 servers, each one have 60 connections to SQL server, how is that an fair comparison?

15. You could easily run 4 instances of WebSphere on p570 in this case, each one with 7GB heap size, ORB theads = 100, web container threads = 200, JDBC connection pool = 100, using EJB mode instead of servlet mode.

Don't conceal facts

===================

The more concealed, the more unveiled. Since the database servers are outside of the SUT and assumed to be unlimited scalable, I have doubt that the .NET StockTrader application is using store procedures or XML processing capabilities of the SQL Server, while I know that the WebSphere Trade application is not doing that. To be fair you should put the database tier in the entire picture for more realistic end-to-end TPS cost.

Summary

=======

IBM System p and commodity blade servers are not designed to run the same exact workload. Granted, there are things that the System p can do that blades cannot, while there are things that blades can do at a lower cost than System p. This is not a comparison of apple-to-apple and anyone can think of a counter-scenario where the blade system is in disadvantage.

The p570 server was ordered with many advanced features that are totally wasted in the proposed setup. No reasonable customers would make this kind of hardware choice and software configuration.

If software price is the only factor to consider, there are many software solutions at no cost, but they won't have the scalability, value-added features and world class support provided by WebSphere and AIX. You could run a story comparing Microsoft Office vs Open Office where only basic features are tested, there's no way MS Office could win in such scenario.

Just a simple glance of the document, I can figure out those flaws, I don't have interest to find more, but if you publish with "Microsoft is responsible for all the content in the document", IBM might have interest...

Oh, by the way, could you please kindly do "splel chcek" for the doc again? If "Database Storeage Specification" on page 13 means "Database Storage".

Response to Virtual Greg (with help from non-Virtual Greg)

Steven B. Martin — Wed, 06 May 2009 18:34:37 GMT

Were consistently asked how high-end RISC-based systems running UNIX (in this case AIX and Power6) perform vs. the ntel/AMD-based servers running Windows Server 2008. A common question we get from WebSphere customers is what is the performance advantage of running WebSphere on high-end RISC based systems such as the IBM Power 570 with a “scale-up” approach, and is it worth the investment? In that sense, this test is simply one comparison – we will do others as well. In this case we are testing a server that is a higher-end system than the Power6 blades you suggest testing, and showing a substantial performance *and* cost advantage of instead running WebSphere (or .NET) on a small cluster of industry-standard servers from HP running Windows Server 2008.

Another interesting comparison would be WebSphere on a mainframe vs. a similar Windows Server configuration. Any comparison is valid, after all, as long as the system configurations and costs are fully disclosed. The cost data is particularly noteworthy - such a huge cost difference in the systems being compared; yet the lower-cost platform is delivering better performance. You are right in that testing other hardware configurations would be valuable as well.

We invested well over $215,000.00 in hardware for the Power 570 server precisely to do these types of comparisons---adding more cores would boost performance but also significantly raise the costs. We would love to have IBM test a 16, 32 or 64-core Power 570 server in these WebSphere tests with us in an independent lab. The BladeSystem from HP can handle up to 16 ½ height blades---we only used 4 blades---so at a cost of roughly $5000 per blade, we could also add significant scale to the HP BladeSystem/Windows Server 2008 setup.

Similarly, a customer might invest in a Power 570 to run enterprise workloads and/or consolidate servers. IBM positions the Power 570 as follows:

• For mid-to-large transaction processing such as ERP and CRM applications

• For mid-to-large database serving

• For server consolidation across UNIX®, IBM i (formerly known as i5/OS®) and Linux® workloads

• For a complete business system combining all aspects of a company’s IT infrastructure

So we are comfortable in the system configurations as tested. But we also provide a tool in the download such that customers can perform their own testing on different servers---Power6-based blades or even WebSphere on a mainframe. Again, these comparisons are valid as well—as long as customers understand both the performance characteristics and the costs of the platforms being compared. In this test, I think we show that running IBM WebSphere on a high-end Power system may not make a lot of sense, considering that system is three-times as expensive and performs less well than a simple cluster of 4 HP blades running WebSphere on Windows Server 2008. Apparently, you agree, and even seem to suggest this is obvious. We do not think it is necessarily obvious to many customers, hence the test and the “Who Knew” campaign. We encourage customers to use the kit to test other hardware/software platforms as well; such as the Power blades you suggest.

Why the 570?

VirtualGreg — Wed, 06 May 2009 00:49:05 GMT

Why test an HP blade against an enterprise server like a 570? The 570 is better suited to large scale server consolidation as a system that supports 32-cores and 3/4TB of memory. Why pit blade against blade and test against the JS22? Why not put your money where your mouth is against the new JS23/JS43? If I read the pdf correctly you also test 8-core POWER6 single system with 32GB of memory against a scale out cluster of 4 blades each with 4-cores and 32GB of memory. 8-core 32GB vs. 16-cores 128GB? Sorry, this benchmark seems a bit tainted to anyone who knows what a 570 is.

...And we'll prove it!

Fri, 01 May 2009 23:59:03 GMT

Yesterday I blogged about some recent findings regarding both system cost and performance when comparing

WebSphere Loves Windows…Who Knew?

SuperSite Blog — Thu, 30 Apr 2009 18:10:18 GMT

Microsoft is taking it's "I'm a PC" campaign to the server and busting some myths in

Websphere on Windows Outperforms Websphere on AIX by 37%, Who Knew?

rickgaribay.net — Thu, 30 Apr 2009 17:39:36 GMT

Websphere on Windows Outperforms Websphere on AIX by 37%, Who Knew?

WebSphere Loves Windows???Who Knew? | Microsoft Share Point

WebSphere Loves Windows???Who Knew? | Microsoft Share Point — Thu, 30 Apr 2009 16:27:18 GMT

PingBack from http://microsoft-sharepoint.simplynetdev.com/websphere-loves-windows%e2%80%a6who-knew/