Inside The Lab

The following is a copy of some objections to this latest comparison, posted by an anonymous source, along with my responses marked by >>>>>:

--------------

Though it is a waste of time to reply a "NO WARRANTIES, EXPRESS OR IMPLIED" document. Rumor stops at wise, I am willing to provide evidence for wise.

It is shameful to see such a highly respected senior personel at Microsoft standing behind such a flawed methodology and benchmark results, and still trying to claim victory. May I ask a few questions about the scenario to clear my mind?

>>>>>Response: Yes!  See our responses below.

Fairness comparison!

====================

1. IBM p570 vs HP c7000 is like using a truck vs 1000 bikes for mail delivery. 1000 bikes will be more efficient and cheaper, but try using 1000 bikes to deliver a nuclear reactor instead of a truck?

>>>>>Response:  This is not a rational argument. IBM would precisely position the Power 570 to run this enterprise workload—and for example consolidate a scale-out cluster of servers to a single box.  This test shows that if you consolidate just 4 (not 1000!) HP blades for any of the 3 workloads tested onto the Power 570, you end up paying three times as much (when using WebSphere on both hardware platforms); and get lower overall performance.  Are you not surprised that just 4 blades from HP (at a cost of roughly $5,000 per) outperforms a system positioned as top-tier Power6 box for server consolidation and WebSphere workloads, at 1/3 the cost of the Power 570?  We have found many customers very alarmed at this; especially in today’s economic climate.  And as for delivering mail vs. a Nuclear Reactor; the Trade workloads are enterprise workloads as defined by the IBM WebSphere performance team themselves; and have been run by IBM on mainframes, Power6 and X86 over the last couple of years in their own published benchmarks—if they cannot perform well on a Power 570 server, you should question IBM, not MS, as to why.  

-----------------------------------------

2. Please don't ignore p570's cost on Reliability Availability and Serviceability (RAS)! The idea of Price Performance Ratio comparison between P570 and HP c7000 misleads readers by ignoring the extra cost on Reliability Availability and Serviceability (or why you ordered AIX service?). It is obviously unfair and even ridiculous! If we find a middle school student who is better in java programming than your senior director, can I get a conclusion that this guy has high price performance ratio?Why order things that are not used in the paper, such as Advanced Power Virtualization, flat panel console, OS maintenance, 2 DVD-RAM drives, etc, while not making equivalent order in the blade system?

>>>>Response: Customers can examine reliability/serviceability  of the two platforms; HP and IBM are best equipped to make their relative arguments on the hardware.  Do not for a second, however, think the HP BladeSystem is not built for both:  in our configuration, we have redundant OnBoard Admin modules; redundant internal network switches in the priced configuration; there is an integrated management/monitoring console for the BladeSystem; even down to monitoring power consumption across blades in the system.  We are using the high-end power supplies/modules from HP; and even invite power consumption comparisons (which we think will favor the HP BladeSystem even more as folks start moving over to the newer Intel Nehalem chips, which we did not use).  As for the “extra” things you believe we priced in the Power6/Power 570 side; this is how the IBM reseller (actually two different resellers came in within $1500.00 using this config; since we got quotes from two, bought from one, and published data from the one we bought from) priced out a std config for us.  Since we breakout the costs---go ahead and remove what you will and then recalculate the costs—you can do this on your own.  You will find the items you list have little impact on the overall $215,000.00 price tag (for example, operator panel = $135.00; or Advanced Power Virtualization per core total cost $7k)….again, this is a typical config IBM resellers are pricing.

---------------------

What is fair then?

>>>>>Response:  Any comparison is fair, as long as there is full disclosure on what is being compared in terms of price and performance.  It is up to customers to decide which comparisons are relevant for their situation.  In this case, for a situation where IBM is trying to sell WebSphere 7 in conjunction with a higher-end Power6 box vs. running on Windows Server 2008 and an HP BladeSystem; the study, we think, clearly shows customers should think twice! As for the other platforms you want to see compared; these would be interesting as well; and why we included in the benchmark kit a Capacity Planner tool that allows customers to fairly easily compare these workloads on any platforms they dare to compare (including WebSphere on mainframe, Power6, or Intel/AMD).  We would love to meet IBM in an independent lab and do several additional comparisons with these workloads; to date they have not accepted our invitation, even though we have offered to cough up the money to pay the independent lab.

--------------------------

3. IBM p570 with AIX vs. HP Integrity with Windows 2008 DataCenter.

4. IBM BladeCenter H vs. HP c7000.

Flaws in cost comparison!

=========================

5. Why order things that are not used in the paper, such as Advanced Power Virtualization, flat panel console, OS maintenance, 2 DVD-RAM drives, etc, while not making equivalent order in the blade system?

>>>>>>Response:  The items you point to amount to a rounding error, at best, when subtracted from the overall $215,000.00 price.  And they are part of the std reseller configuration/IBM recommended configuration, after all.  Since we breakout the costs separately, any customer can remove items they deem unnecessary, and look at how little impact they have on the overall cost of this particular server (Power 570).

--------------------

6. HP hardware price: why no Ethernet switch in the chassis, no console kit, etc?

>>>>>Repsonse:  There are two redundant internal Ethernet connect modules priced in the HP configuration, as documented in the paper, these are in the chassis (but are pluggable in back)—just as you suggest.  As for console kit; there is an integrated flat panel console in the HP BladeSystem; and the capability to connect via Web to the onboard admin console from any web browser; or use serial/modem.  But go ahead, if you want, and remove the trivial cost for the Power 570 operator panel; it will make little difference to overall price; but also know IBM Resellers would doubtfully ever sell such a config on the Power 570 side.

---------------

7. WebSphere AppServer price includes support, .NET solution has none.

>>>>>Response: This is the standard pricing for WebSphere Application Server; they include some level of support via Passport Advantage subscription (no additional higher-end support was selected for either .NET or WebSphere).  The included support is largely web-based and ability to download product patches/fixes; etc---which you get with .NET for free.   If customers want to price MS Premier Support for .NET vs. IBM’s premier support offerings for WebSphere, they can do so; on the MS side visit http://www.microsoft.com/services/Microsoftservices/srv_compare.mspx  for a chart/comparison of MS enterprise support options.

Also note IBM would likely recommend for enterprise customers (or try to sell) WebSphere Network Deployment Edition, which is actually over three times as expensive as the WebSphere App Server edition we priced---so keep this in mind as well.  Even with discounts, most IBM WebSphere shops are likely paying even more for their WebSphere licenses!

------------------------

Misleading in Test bed architecture:

====================================

8. Why using only 1 instance of WAS against 2 database servers? It's not possible to make use of both of them.

>>>>>>Response:  As documented in the paper, two instances of WebSphere were run on the Power 570 for the two workloads involving database access.  We also ran four instances; with no overall impact on throughput; two was enough.  But, you can certainly use the provided tools to test the workloads against as many WebSphere instances as you want.  

------------------

9. It would be fairer to run 4 instances of WAS on p570, distributing SQL workload to both database servers equally.

>>>>>Response: We found two vs. four had no difference on the Power 570.  One instance going to one DB; the other instance going to the other DB.  You can use the provided Capacity Planner to test other configuration; we are confident you will not find any significant difference.  Overall, we spent about 6 months on this benchmark—part of this inclusive of moving from WebSphere 6.1 to WebSphere 7 with required Java code changes to target new IBM JVM based on newer Java Enterprise platform; and much of it testing a wide variety of configurations and tuning settings for WebSphere to ensure we got optimal results for WebSphere 7.  As pointed out below, this included testing different heap sizes as well.

-------------

10. The p570 has 8 cores, 32GB memory and 4 NICs, while the c7000 has 16 cores, 128GB memory and 8 NICs combined. They are not equivalent hardware or architecture, but the wide difference tells a lot.

>>>>>Response: Network utilization remained below 25% on each NIC in the Power 570 server. As for memory, WebSphere 7 was consuming ~4GB total (we tested with smaller and larger heap sizes; as long as heap is above about 1.4GB, no difference since none of these workloads are memory intensive—try for yourself!).  The 4GB consumed on the Power 570 leaves *plenty* for AIX/system level resources; memory consumption never approached even 50% of available on AIX (or Windows Server for that matter, where each blade had the same 32GB memory as AIX/Power 570).  As for more cores—you are right!  Comparing just raw performance on a per core basis; you can see the Power6 chips at 4.2GHz do outperform the Intel 3.0 GHz XEON chips.  The point is scale up vs. (moderate) scale out; and the huge price differential in the configurations.  We are eager to test on newer Intel Nehalem chips in the future; and compare additional configurations.  Customers should accelerate their move from high-end RISC to industry standard Intel/AMD hardware, we believe this test clearly shows why.

-----------------------

Misleading in Middleware optimization

=====================================

11. EJB is heavier for simple tasks, but better for high volume workload such as this one. Why is servlet mode being used for WebSphere Trade on p570?

>>>>>Response: Having tested Trade 6.1 in all modes, including on newer EJB 3.0/JPA architecture (which required significant code changes moving from WebSphere 6/CMP 2.1 to WebSphere 7/EJB 3; in all comparisons straight JDBC/servlets outperformed EJB modes.  Take a look at a past comparison, for example, on different hardware:  http://media.techtarget.com/TheServerSideNET/downloads/DotNet-WebSphere_Web_Services_Benchmark.pdf .  Did you know EBay (as presented at SD West about two years ago, got rid of EJBs in order to achieve scale---they recommend “•  J2EE: use servlets, JDBC, connection pools (with rewrite). Not much else.”  You can see for youself at:

http://highscalability.com/ebay-architecture

So we tested with what we know, based on massive numbers of tests for these workloads over three+ years, to be the fastest WebSphere configuration for this app. You of course are free to test other implementations, using the provided Capacity Planner multi-agent benchmark tool provided in the kit (with all source).  We would gladly accept from IBM precise feedback on tuning parameters/modes tested, and re-test/re-publish if we find ourselves to be wrong; but IBM needs to provide precise tuning/server.xml file + Trade configuration to test for the hardware SUTs as tested.

-------------

12. Setting WebSphere on p570 with 32GB memory to run at max heap size of 2GB, why waste the remaining 30GB? You could run 14 of such instances concurrently!

>>>>>>>Response: We tested heap sizes ranging from 1GB to 4GB; 2GB was plenty for these workloads; you do not want to go too big if not needed; as you know performance of GC tails off as GC has to collect across larger and larger heap sizes.  Again, we are confident in the results and the massive number of tests/settings tried that we got this right—both in WebSphere instances and heap sizes.  You are free to try different heap sizes/WebSphere instances if you want, with the provided Capacity Planner tool and all sources published in the kit.

-------------------

13. Setting Trade to run at servlet mode with maximum web container threads at 50 on a 8-core p570? It means you're using 8 cores to run only 50 threads max, while this system was designed to accommodate thousands of concurrent threads.

>>>>>Response:  We tested from 25 to 100 Web container threads, going up by ten in different optimization runs.  In WebSphere 7, 50 is the new default for Web Container threads; we found this setting (with two instances running) to be fine when running the Trade workloads with many thousands of concurrent clients. Lower thread settings below 40 impacted in negative way (connections refused); higher had no impact.  We are confident the 50 thread setting produced peak TPS possible (remember we are running two instances in addition, on the two DB-backed workloads).  This setting is also very close to SPEC JappServer results/tuning settings IBM uses in their benchmarks.  At any rate, we are very confident given iterative tests and fact we got to near 100% CPU saturation of the Power 570 the tuning settings we used produced (as close as possible) peak TPS rates on the Power 570.  

-------------------

14. On the p570 where only 1 WebSphere instance is run, setting maximum JDBC connection pool to 51 while each database server can accept 400 connections, what's the point of this limitation? In the .NET scenario there are 4 servers, each one have 60 connections to SQL server, how is that an fair comparison?

>>>>>>Response:  Again, we are confident in these settings; but customers can try others if they wish---or IBM can recommend a specific set of settings and we will happily try these and publish our results!  In this case, all DB connections are used by Web Container threads---we set JDBC connection pool size to be one bigger (one free connection at all times) than Web Container Threads; plus remember we are running two WebSphere instances on the Power 570, as documented.  You will not get better results with more threads or more DB connections in the pool; as we again performed hundred of iterative runs. Interestingly, if you set lower number of JDBC connections in pool than Web Container threads and run high load, you will hard crash WebSphere!  Even with newer WebSphere 7; this has been true for every edition of WebSphere we have tested beginning with 3.x----it does not properly queue up requests against the JDBC connection pool.  ADO.NET does; we set for 60 on ASP.NET side; thread pool on ASP.NET auto-configures itself—its not a hard number.  Incidently, there is a huge difference between time it takes to tune WebSphere apps for best performance; vs. typical .NET application where you do not need to configure heap sizes; and typically not even thread pool sizes since these auto-tune.  But again, don’t think these tests were done in a hurry---we took lots of time to get it right, have lots of experience with these workloads across WebSphere editions, and .NET, of course.  That is why we encourage customers to test for themselves; and why we are confident overall that IBM themselves would not get better results as tested.  We would love to meet IBM in an independent lab, as we mentioned before, to test these workloads on this and other hardware configurations, with full pricing disclosure.

----------------

15. You could easily run 4 instances of WebSphere on p570 in this case, each one with 7GB heap size, ORB theads = 100, web container threads = 200, JDBC connection pool = 100, using EJB mode instead of servlet mode.

>>>>>Response:  You will not find EJB mode (even once migrated to EJB 3.0) perform better than JDBC mode as long as you run as we did, with no stale data (real time data from DB) except for Market Summary where both WebSphere (via Servlet Caching) and .NET (via Asp.NET output caching) cache this business object for same time (180 seconds absolute).  But you can test for yourself; of course.  We are confident in the results we published, with caching equalized across all platforms and no stale data for stock quotes, account balances etc; are the fastest you can get on WebSphere for Trade app; and we are confident the caching policy we chose is the most real-world for this application.  You can play a lot of tricks with EJBs and WebSphere cache, to be sure (as you could with .NET as well); but as long as you want the business objects to reflect what is actually in the DB, and you want the DB accessible/updatable from other apps or even other WebSphere clusters; you cannot just play these caching tricks/EJB tricks without possibility of wrong/corrupt data being presented/stored.  Again, one of the classic arguments for why you DO NOT want to punt database concurrency to the middle tier----you want to rely on the database (SQL, Oracle, DB2) since this is what they are designed to do; but when possible design a realistic cache policy with respect to data display, where some data staleness is ok.

-------------

Don't conceal facts

===================

The more concealed, the more unveiled. Since the database servers are outside of the SUT and assumed to be unlimited scalable, I have doubt that the .NET StockTrader application is using store procedures or XML processing capabilities of the SQL Server, while I know that the WebSphere Trade application is not doing that. To be fair you should put the database tier in the entire picture for more realistic end-to-end TPS cost.

>>>>>>>Response:

The SQL Server database is using neither stored procs or SQL Server XML processing capabilities.  It is configured exactly the same as the DB2 database, using dynamic SQL queries, full tx logging, and running on exact same hardware.  Download the code and see for yourself. This is a middle tier test; not a DB test; however, since the middle tier DB drivers do come into play, we opted to use IBM’s recommended database for WebSphere (DB2) and MS’s recommended database for .NET (SQL Server)---seems only fair; unless IBM wants to argue that Oracle or SQL Server JDBC thin drivers are better/more efficient than DB2 JDBC thin drivers (and we know they would never make this claim).  You can easily price out the software costs for the database if you desire using vendors’s published suggested retail pricing/IBM Passport advantage pricing for DB2 and MS published pricing, given we fully disclose the hardware its running on.  However, we opted not to include DB pricing since the DB is not part of the SUT---it is supersized on purpose such that it in no way impacts top-end TPS results for the middle tier on WebSphere/AIX; WebSphere/Windows; or .NET/Windows.  The pricing data as published is all about the middle tier (for which there might be many such servers in an enterprise operating against a common database).

-----------------------------

Summary

=======

IBM System p and commodity blade servers are not designed to run the same exact workload. Granted, there are things that the System p can do that blades cannot, while there are things that blades can do at a lower cost than System p. This is not a comparison of apple-to-apple and anyone can think of a counter-scenario where the blade system is in disadvantage.

>>>>>Response:  Really?  Please be more clear because in our enterprise customer encounters, we find IBM selling or attempting to sell everything from mainframes to high-end Power6 systems for precisely these types of WebSphere workloads!  After all, they make more money on the hardware and more money on the WebSphere software to boot when selling on Power 570 (or even Power6 blades) vs. Intel-based blades.  Please show us IBM published information that backs up your claim or provides detail in what one platform can do vs. the other!  This seems wishful thinking and the typical cry of “its not fair !” when you do not see the results you expect.

-------------

The p570 server was ordered with many advanced features that are totally wasted in the proposed setup. No reasonable customers would make this kind of hardware choice and software configuration.

>>>>Response:  Really?  Please contact two or three IBM resellers and ask for an 8-core Power 570 server and see how they configure it!  Or, even; remove *all* the items you complain about (they are priced by component in the paper so you can do this); subtract the costs of these items, and come back with what your non-standard, 8-core Power 570 configuration would still cost. It will be well above the cost of the WebSphere on Windows Server 2008/HP Blades, to be sure; as the items you want removed are trivial given the over $215,000 cost of just the hardware/AIX OS alone.

-------------------

If software price is the only factor to consider, there are many software solutions at no cost, but they won't have the scalability, value-added features and world class support provided by WebSphere and AIX. You could run a story comparing Microsoft Office vs Open Office where only basic features are tested, there's no way MS Office could win in such scenario.

>>>>>Response:  Doesn’t this test show the scalability of WebSphere 7 on both Windows Server 2008 and Power6/AIX is very good?  The point is; after looking at the relative performance data, you must also look at the cost data for the scale-up Power 570 approach vs. the WebSphere on Windows Server 2008/HP BladeSystem approach.  And, we will add, look at the enterprise customer case studies for .NET at http://www.microsoft.com/windowsserver/mainframe/whoknew/  and the overall widespread adoption of .NET and you will see that Windows Server 2008 does a great job in terms of scale and performance for both .NET and WebSphere/Java apps.

--------------------

Just a simple glance of the document, I can figure out those flaws, I don't have interest to find more, but if you publish with "Microsoft is responsible for all the content in the document", IBM might have interest...

>>>>>>Response:  We are responsible for all the content; we stand behind it and even offered to meet IBM in an independent lab to test a Power 570 server vs. WebSphere and .NET on Windows Server/HP Blades. No response.  We would be doing customers a grave injustice and hence harming ourselves if we published bad data.  Hence the full disclosure document; the code; and even the Capacity Planner tool customers and IBM can use for themselves to verify the results; and our challenge to IBM to meet in an independent lab to perform these tests.

----------------------

Oh, by the way, could you please kindly do "splel chcek" for the doc again? If "Database Storeage Specification" on page 13 means "Database Storage".

>>>>Response:  Thanks, will do!