TPC-E – Raising the Bar in OLTP Performance

re: TPC-E – Raising the Bar in OLTP Performance

kirchner — Fri, 24 Oct 2008 14:02:31 GMT

I totally agree with you.

TPC-C is too old and proves nothing but the fact we can attach about 30 thousand disks to a single server. It's a very unrealistic configuration and absolutely won't reflect our real world.

It may be the fact that other vendors can't beat SQL Server (after all, what's faster and better than SQL Server apart from itself???), but I personally believe that people are still giving too much credit on TPC-C.

While customers don't realize TPC-E is FAR BETTER and start questioning "why my database vendor has not published any results?", vendors won't feel the pressure and won't leave their confort place.

re: TPC-E – Raising the Bar in OLTP Performance

ilovedatabases — Sat, 08 Nov 2008 02:02:40 GMT

Charles, we've met - you're a smart guy..I'm somewhat shocked at the posting. But I commend you on the work you are doing on TPC-E - indeed, it will become the standard, when it becomes the de-facto standard. And yes, I grow tiresome of all the disks in TPC-C too. And I work on all databases....but to make it interesting, the community has to support it.

(1) One thing about Mr. Paulley is that indeed he is hitting the point home about an optimizer. Indeed, not everyone writes SQL that is succintly aware of the underlying engine like folks in the MS R&D lab do. So to me, all those developers out there should write a T-SQL statement and the database should do what it can to rewrite it and make it faster. Having a guy on your performance team write fast SQL for his database isn't that impressive to me.... cause I can't hire him.

(2) What about the quesiton why isn't Microsoft also pursuing TPC-C with TPC-E? In fact, after saying no TPC-C for SS08, MS released a TPC-C for 05. AFTER. We're talking in the last couple of month on a hexa core. If it is so bad, why did you folks do it? If you look at the per core output of top results, MS has long had some issues in TPC-C area. In fact, I seem to even recall an apples to apples comparison a # of years ago with a competitor on the windows platform and same hardwrae (very rare) that beat SQL Server 2005 by 15%! So why the bi-polar TPC-C and TPC-E if this is so true? MS was never going to find a winning per core sustainable mark cause of platforms like POWER and thus WISELY invested early in TPC-E for good marketing...but when it all catches up, what is the answer.

(3) You compare TPC-C and TPC-E in a chart -- but notty notty - you can't do that.

(4) You talk about a non-RAIDed TPC-C result (and that is sneaky I admit) but somehow missed that MS's biggest 1 TB TPC-H result for SS05 launch didn't raid the disks either.

(5)You make it sound like MS doesn't play tricks in TPC-C such as pricing in a single call for support, turning off statistics collection, and some vendors even turn off integrity checks of the page. The point is MS or Oracle when they publish have all kinds of games no one ever does in the real world; if I pull the full disclosure for TPC-E will I find some there (I haven't, but don't make me read 400 pages for nothing..lol)

(6) Why do I like TPC-C -- NOT because of the results. Because all things being equal, it stresses out the database logger. SQL Server 2005 logs almost 3 times more KB than some other solutions per TX in TPC-C; I'm not sure that is a metric dislosed in TPC-E (it wouldn't be that great since the SQL wouldn't be the same) but it certainly tells me something when everyone is doing the same thing.

Now I do agree with you at the end, any benchmark publication is a lot of work, and you tune towards it. Do you really believe Teradata can't scale since they don't publish TPC-H? They feel it a problem solved (I say they don't like to disclose costs)

So you say vendors can't beat existing results; is that they will never or haven't spent the time investing in that area and when they do they won't be able to.

It's dangerous words for a person of your reputation. TPC-C still has some validity (logger stress on equal footing) and TPC-E has some great modern stuff to.

But I'd prefer a non attempt at marketing I'm a developer post :)

As for Kirchener above to say nothing is faster than SQL Server -- well..I won't waste the characters on that one. I like SQL Server ,please don't get it wrong. I don't like when smart people step into the foggy world of FUD- I would expect this from Oracle!

re: TPC-E – Raising the Bar in OLTP Performance

databasemysterio — Sun, 09 Nov 2008 15:41:06 GMT

I like to hear more on TPC-E -- but I agree with above on FACTS not FUD. BOTH MS and IBM (referenced above) use RAID0 in TPC-C so I find the comment disingenuous.

AS well, TPC-C isn't just too many disk test -- above someone hits on loggings, it's also SMP scale-up, concurrency, memory management, and more. So it that regards, even if the #s don't impress you, looking at that stuff does.

Finally, solid state drives are going to change disk layouts.

But the biggest reason I'd like MS to post TPC-C is that Oracle still does it and is planning another one from what I see in blogs.

Either way, I'm cool - I just like GOOD information -- not marketing FUD from R&D -- let marketing do that I already bought Vista.

re: TPC-E – Raising the Bar in OLTP Performance

glennpaulley — Mon, 10 Nov 2008 19:32:00 GMT

I've posted a response to your remarks on my blog.

re: TPC-E – Raising the Bar in OLTP Performance

Data & SQL Storage Performance Team — Thu, 20 Nov 2008 02:47:56 GMT

Raid vs. non-RAID

Ilovedatabases implies that it is disingenuous on my part to criticize TPC-C results for using non-RAID protected storage while at the same time having TPC-H results that use unprotected storage. This misses my point, which is that it is the benchmark specification itself that establishes what level of storage protection is required. TPC-E requires storage protection; TPC-C and TPC-H do not. It’s perfectly fair for vendors to play to the minimum requirements of the benchmark. The problem is when those minimum requirements are flawed or outdated. Real customers don’t run mission critical OLTP systems with unprotected storage. And yet that is what TPC-C requires and how everyone runs it. TPC-H is also outdated in this regard, but there’s not an alternative benchmark in the DW space.

Playing by the Rules

Ilovedatabases says everybody plays tricks when they run the benchmarks. Of course, one man’s trick is another man’s feature. Everyone tries to satisfy the requirements, but nothing more. Over time, everyone gets closer and closer to the edge. This is why it is important to move the requirements higher. The only way to do that is by defining a new benchmark, e.g., TPC-E.

Fighting the Last Battle

Ilovedatabases says he (she?) likes TPC-C because it stresses the transaction log. TPC-A stressed logging even more than TPC-C does. You had to do write ahead logging (WAL) to be competitive. Anyone who didn’t already have WAL implemented it. But that’s water under the bridge. Nobody is going to take it out. So it’s not a good reason, IMHO, to favor TPC-C (or TPC-A).

TPC-C results still being done

Simply looking at the number of published results on TPC-C versus TPC-E doesn’t tell the full story. (In reference to Glenn Paulley’s response.)

Microsoft doesn’t directly publish TPC benchmark results. Our hardware partners (e.g., HP, Dell, IBM, Unisys, NEC, etc.) publish results. We provide them a kit and support. Our level of direct involvement in publication of any particular benchmark result can vary greatly from no involvement whatsoever to actively working with the partner throughout the benchmark process.

At the launch of SQL Server 2008 (in February 2008), Microsoft announced that we are shifting our focus to TPC-E and away from TPC-C. We have not and will not authorize any TPC-C results on SQL Server 2008 or later versions. To accommodate our partners’ needs, we have allowed them to continue publishing TPC-C results using SQL Server 2005, while encouraging them to transition to TPC-E. All of the TPC-C results using SQL Server 2005 published this year have been run entirely by the hardware sponsors with no involvement or encouragement from Microsoft.

There are more TPC-E results published year-to-date than TPC-C, even though TPC-C results span all database vendors while TPC-E results are (so far) only on SQL Server. I expect this trend to continue with TPC-C results becoming fewer and further apart as the value of legacy comparisons declines over time.

-Charles Levine