SQL Server Performance : Announcements

SQL Server is Movin' On Up!

Data & SQL Storage Performance Team — Mon, 10 Nov 2008 21:34:00 GMT

At last week's WinHEC in Los Angeles, SQL GM Quentin Clark joined Windows Server VP Bill Laing on stage, to announce our upcoming support for more than 64 logical processors, the current limit in Windows and SQL Server. This expanded scale-up capability is planned to be released in SQL Server code named "Kilimanjaro", when running on top of Windows Server 2008 R2.

We expect to support up to 256 logical processors in this release, though that's a soft limit. The hard limits are much higher than this, but we won't support what we can't test. You'll likely see this 256LP soft ceiling get higher in each release, now that we've finally done the heavy lifting of raising the roof on Windows and SQL Server.

The keynote demo (@33:26), which Bill and Quentin did together, went well. Many thanks to our friends at HP and IBM for all their support and for the use of these great servers.

In case you're wondering, a logical processor is a sub-unit of a physical processor/socket/package. Typically today an LP means a core, but it can also mean a hardware thread (nee hyper-threading).

-David Powell

SQL Server 2008 is on its way with great performance

Data & SQL Storage Performance Team — Wed, 06 Aug 2008 20:01:00 GMT

Just left the final shiproom meeting for SQL Server 2008, and am happy to say every team has signed off, so the product is now in the hands of manufacturing, and in process toward web and media availability for you. We've shipped! MSDN and TechNet subscriber downloads are now live, with more to come.

It's been less than three years since SQL Server 2005, but we're pleased to give you a great new release of SQL Server which not only adds fantastic new capabilities to your data platform, but also delivers broadly better performance. We used new industry standard benchmark workloads, and customer workloads, to drive us toward delivering better real-world performance...and we didn't take the easy road.

The best example of that is our use of TPC-E, a far more modern, realistic, and challenging benchmark workload than its predecessor. We used TPC-E to improve the scalability of our relational engine, in ways that should be more relevant to your own OLTP database workloads. We're proud of partners, like IBM, NEC, and Unisys, who were able to demonstrate great scalability, up to 64 cores, using SQL Server 2008 running TPC-E, and like Fujitsu-Siemens and Dell, who have shown industry leading price/performance. Results like these should also be more useful for system sizing.

Almost five years of effort by TPC members were invested in the development of TPC-E, and it shows. The workload uses synthetic data which is far more realistic, by modeling real-world data. And compared to its antique predecessor, TPC-E's schema has ~3X more tables and primary keys, 2X as many columns, and 4X more foreign keys. And here's a radical thought for a modern OLTP benchmark workload: include check constraints, referential integrity, and reliable storage. Don't customers actually put DBMS servers into production expecting that?

Because we believe so strongly TPC-E drives us to better meet your needs, I am announcing today this is the first release of SQL Server which will not include published TPC-C benchmark results. Like other great thoroughbreds, TPC-C had a great run, and we were proud to ride it while it was still relevant to customers. But today, we're turning that great old race horse out to pasture for a well-deserved rest.

In SQL Server 2008, we have also invested lots of effort to improve our data warehousing performance, and the performance of our BI services. SSIS, SSRS, and SSAS, each show many double-digit gains in performance, which we hope you will enjoy. The new world record we set with SSIS, loading 1 TB of data in under 30 minutes, gives you a sense of this commitment to BI performance. We are also proud of our first-ever TPC-H 10 TB result.

There are a couple other ways in which this release improves on our past work. First, we've focused more energy on improving 64-bit SQL Server's performance on x64 AMD Opteron and Intel Xeon architectures. Given the price and density of RAM, and great new x64 CPUs available, this is a perfect time to take a closer look at the performance of your database servers, and consider doing fresh deployments, or migrations, on x64 SQL Server, which can use the additional RAM for everything, not just the buffer cache. Second, we invested more heavily in performance regression testing, both in automation and in the breadth of our test coverage. These investments have been reflected in broadly positive feedback from the community as well as internal and external beta sites. While we'll always have more to do, we feel this release marks an important step forward.

Welcome to the beginning of a new era for SQL Server. We hope you enjoy working with SQL Server 2008 and look forward to your feedback.

-David Powell
SQL Performance Engineering

ETL World Record!

Data & SQL Storage Performance Team — Wed, 27 Feb 2008 22:36:00 GMT

Today at the launch of SQL Server 2008, you may have seen the references to world-record performance doing a load of data using SSIS. Microsoft and Unisys announced a record for loading data into a relational database using an Extract, Transform and Load (ETL) tool. Over 1 TB of TPC-H data was loaded in under 30 minutes. I wanted to provide some background material in the form of a Q&A on the record, since it’s hard to give many details in the context of a launch event. We are also planning a paper that talks about all this, so think of this article as a place-holder until the full paper comes along. I hope you find this background information useful.

- Len Wyatt

How fast was the data load?

More than one terabyte of data was parsed from flat files, transferred over the network and loaded into the destination database in less than 30 minutes, a world record beating all previously published results using an ETL tool. That is a rate in excess of 2 TB per hour (650+ MB/second). To be precise, 1.18TB of flat file data was loaded in 1794 seconds. This is equivalent to 1.00TB in 25 minutes 20 seconds or 2.36TB per hour.

Why is this important?

Businesses have ever-increasing volumes of data stored in many heterogeneous systems. Thay want to know that an ETL tool they choose will be able to support any data volumes they might require. Microsoft has been making a significant investment in SQL Server Integration Services (SSIS), and this record illustrates the capability of SQL Server Integration Services 2008, SQL Server 2008 and the Unisys ES7000 to handle a significant volume of data at a dramatic speed.

Why not just do a bulk load of the data?

It is rare in businesses today that data is always available on the destination system, and does not need to be standardized or corrected for errors before loading. These rare cases are the times that bulk loading data makes sense. Data integration can involve complex transformation rules, error checking and data standardization techniques. ETL tools like SSIS can perform these functions such as moving data between systems, reformatting data, integrity checking, key lookups, tracking lineage, and more. SSIS has proven itself to be a versatile ETL tool, and now it is shown to be the fastest one as well.

What data did you choose to load?

DBGEN tool from the TPC-H benchmark was used to generate 1.18 TB of source data. The data were partitioned by DBGEN, allowing it to be loaded in parallel from multiple systems. DBGEN generates data on customers, parts, suppliers, orders and line items. It is broadly representative of a wholesale business. The data contain a variety of data types, including dates, money amounts, integers, strings and flags.

Please note that the ETL loading results are not TPC-H benchmark results and should not be compared to TPC-H benchmark results.

Was this a certified benchmark?

There is no commonly accepted benchmark for ETL tools. Microsoft thinks there should be. Industry standard benchmarks can lead to healthy competition, better products, and better publication of the techniques used to get high performance. Microsoft would welcome the opportunity to join with others in the industry to define a common benchmark that reflects the real-world uses of ETL tools.

The use of TPC-H data for this project was a convenience. This is not a TPC-H benchmark result.

How does this compare to your competitors?

Multiple competitors have published results based on TPC-H data. Informatica has the fastest time previously reported, loading 1 TB in over 45 minutes. SSIS has now beaten that time by more than 15 minutes.

There are other claims of fast times that have been made, but on non-standard data sets and without enough information to allow any meaningful comparison. This is part of the reason Microsoft would support the creation of an industry standard ETL benchmark.

What system configuration was used?

The database server ran on a Unisys ES7000/one Enterprise Server , with 32 socket dual core Intel® Xeon^TM 3.4 Ghz (7140M) processors , 256 GB RAM and 8 dual port 4Gbit HBA’s . The SQL Server data was stored on an EMC Clariion CX3-80 SAN with 165 (146 GB/15 krpm) spindles. The database server ran a pre-release build of SQL Server 2008 Enterprise Edition (V10.0.1300.4, built just before the “February 2008 CTP”) on the Windows Server 2008 x64 Datacenter Edition operating system.

Four servers acted as data sources, modeling the fact that data comes from a variety of systems in a modern enterprise. Each source server ran SSIS packages that sent data across the network to the database server. The source servers ran SSIS from SQL Server build V10.0.1300.4, on the Windows Server 2008 operating system. Source data came from flat files, as it was generated by DBGEN.

For the source servers, 4 Unisys ES3220L servers with Windows2008 x64 Enterprise Edition were used. Each server is equipped with 2 x 2.0GHz quad core Intel® processors, 4GB RAM, a dual port 4Gbit Emulex HBA and Intel PRO1000/PT network card. The source data was read from 2 x EMC Clariion CX600 SAN’s with 45 spindles each.

The Source servers were connected to the ES7000/one server database server with private dual port 1Gb Ethernet connections.

Why use multiple source systems?

Modern large businesses are complex operations. Large data sets are often the result of multiple data feeds. This made the test more realistic by mimicking a real world ETL scenario.

What do the SSIS packages look like?

There was just one package, though the source systems ran multiple instances of it. It is quite simple: There is one control flow for each “stream” of data generated by DBGEN. The control flow has one data flow for each table, each data flow reading data from a flat file source and writing to the SQL Server database via OLEDB. Using this data set there is a one-to-one column mapping between the flat file data and the database tables.

Did Windows Server 2008 figure in to this?

A lot of innovative engineering work in Windows Server 2008, including significant improvements in memory management, PCI and block storage I/O, and core networking, helped achieve this great performance. Because of these advances, Windows Server 2008 sustained about 960 megabytes per second over the Ethernet network, during processing of one large table.

Were secret internal tricks were needed to make this work?

No secret internal tricks or special builds were needed. Although this project used a pre-release version, it was a regular SQL2008 Enterprise Edition build. No special code in the product was used. Everything we did could be replicated by others.

The main thing done in the relational database was to use “soft NUMA” and port mapping to get a good distribution of work within the system. This is a published technique; you can find articles about it on MSDN. We also set the –x flag on starting SQL Server. This reduces the time SQL Server spends collecting performance statistics at run-time.

In SSIS we made sure the data types used in the SSIS data flows matched the types used in SQL Server, so the data did not need to be converted again after the initial conversion of strings read from flat files. Fast Parse is set on the text file fields where it applied.

The network connections on the server used the built-in Intel PRO/1000 GbE controllers. Released versions of network drivers were used, and Ethernet jumbo frames were configured to better support this bulk streaming scenario. Window Server 2008’s new TCP/IP receive window autotuning was set to “restricted”. The IntPolicy tool was used to ensure the ES7000 server NICs’s interrupts & DPCs occurred on a CPU affinitized to the same NUMA node as the NIC.

A complete list of settings and optimizations will be included in the paper when it is released.

SQL Server 2005 SP2 Re-release and post fixes

Data & SQL Storage Performance Team — Thu, 12 Apr 2007 00:20:55 GMT

Bob Ward from PSS has a wonderful blog article that explains the details about the re-release of SQL Server 2005 SP2 and fixes posted later. This is a must read for anyone deploying SQL Server 2005 SP2 to understand the various hotfixes, GDRs and procedures. Please visit his link for more details.

SQL Server 2005 SP2 Re-release and post fixes

-- Umachandar

New MSDN Books Online search functionality

Data & SQL Storage Performance Team — Fri, 30 Mar 2007 21:52:55 GMT

Check out the new Books Online search functionality online. The link below provides a scoped search of Books Online that returns a more precise and targeted result set. You can use it to search Books Online content quickly.

http://search.live.com/macros/sql_server_user_education/booksonline

Please direct your feedback to SQLServerUE@hotmail.com.

-- Umachandar

SQL Server 2005 Performance Dashboard Reports

Data & SQL Storage Performance Team — Thu, 29 Mar 2007 22:23:24 GMT

The SQL Server 2005 Performance Dashboard Reports are Reporting Services report files designed to be used with the Custom Reports feature introduced in the SQL Server 2005 SP2 release of SQL Server Management Studio. The reports allow a database administrator to quickly identify whether there is a current bottleneck on their system, and if a bottleneck is present, capture additional diagnostic data that may be necessary to resolve the problem. For example, if the system is experiencing waits for disk IO the dashboard allows the user to quickly see which sessions are performing the most IO, what query is running on each session and the query plan for each statement.

Common performance problems that the dashboard reports may help to resolve include:
- CPU bottlenecks (and what queries are consuming the most CPU)
- IO bottlenecks (and what queries are performing the most IO).
- Index recommendations generated by the query optimizer (missing indexes)
- Blocking
- Latch contention

The information captured in the reports is retrieved from SQL Server's dynamic management views. There is no additional tracing or data capture required, which means the information is always available and this is a very inexpensive means of monitoring your server.

Reporting Services is not required to be installed to use the Performance Dashboard Reports.

Check out these reports and experiment with them to troubleshoot your performance problems. You can find the reports at:

SQL Server 2005 Performance Dashboard Reports

Umachandar

SQL Server 2005 SP2 has been released...

Data & SQL Storage Performance Team — Tue, 20 Feb 2007 01:23:34 GMT

SQL Server 2005 Service Pack 2 has been released. You can find more information from the service pack 2 page.

This service pack extends the functionality of SQL Server in many ways. The What's New document highlights the major changes in the Database Engine, Analysis Services, Integration Services and others. The service pack also has performance and manageability enhancements that will be of interest to lot of developers and database administrators.

Please download the latest service pack and test your database applications with it to see the benefits.

Umachandar Jayachandran

Database Applications Profile Survey

Data & SQL Storage Performance Team — Sun, 04 Feb 2007 22:11:00 GMT

We are conducting a survey to better understand SQL Server workloads.

Please consider providing your workload characteristics and hardware configuration to help us better understand the workloads that run on SQL Server. This is a detailed survey so we are hoping to determine more than what SQM or SQLH2 or other mechanisms offer today. Thanks for participating.

Umachandar Jayachandran

Welcome!

Data & SQL Storage Performance Team — Thu, 25 Jan 2007 03:28:33 GMT

Welcome to the new bloggy home of Microsoft’s small but mighty SQL Performance Team!

Lots of smart folks on the SQL team work on performance, as some part of their job. We do it every waking moment. We live and breathe performance & scalability, across the box, at a code level and at a system level. The components we study and improve range from developer technologies that live in .NET to our great BI products to the core engine components.

We benchmark to understand performance issues and to demonstrate where we stand. We develop prototype solutions and we work together with product development teams to engineer performance improvements. We also build systems for performance assurance. High-quality performance proof points are valuable to our customers, so we also participate in the TPC benchmark process, together with our partners. The right benchmarks drive the industry to improve products to better meet customers’ needs.

We dream of every Microsoft SQL customer smiling because they don’t worry how their database, applications, and tools, perform. We are working hard to make sure each release improves in scalability, and delivers performance that is more predictable and dependable than the last one.

Over time, I hope you’ll find the information shared here interesting, at times entertaining, and always worthwhile enough to earn a few minutes of your reading time each week. We also hope you’ll not be shy in sharing with us your experiences and views on SQL performance.

-David Powell