Microsoft runs its worldwide operations on SAP ERP. With 93,000 employees and operations in more than 110 countries, the largest software company in the world runs some 25 million SAP transactions a month against its 5.8-terabyte compressed database. The company was happy with how well its SAP deployment was performing on earlier versions of SQL Server and Windows. Nevertheless, it upgraded to the beta edition of Microsoft SQL Server 2012 to take advantage of new features, specifically AlwaysOn. Leveraging AlwaysOn eased administration and aligned the processes for local and remote high availability. This made the disaster-recovery process easy because the same overall process is being used. At the same time, potential data loss in a disaster case could be minimized.
Microsoft has relied on SAP ERP software to run its financial operations since 1996 when it first deployed the solution on Microsoft SQL Server 6.5 data management software. Since then, the Microsoft SAP ERP deployment has accumulated a 5.8-terabyte data volume in the back-end database despite a rollout of SQL Server database compression. When the company upgraded to SQL Server 2008 R2 in 2009, the monthly growth rate dropped from more than 200 gigabytes (GB) to around 120 GB despite more than doubling the number of business transactions through the SAP ERP system over the last three years. Besides SAP ERP, Microsoft runs another eight SAP NetWeaver applications, which perform distinct parts of business processes.
With 93,000 employees, operations in more than 110 countries, and 2011 revenues of U.S. $70 billion, Microsoft has a huge volume of financial and operational data to track. The company’s SAP ERP system handles its treasury; worldwide sales, finance, human resources, and operations; material management; U.S. payroll; and most other mission-critical functions.
SAP ERP performance at Microsoft includes:
More than 1,900,000 dialog steps per business day.
66 million transactions per month (79 million peak during quarter-end reporting).
An average user response time of less than one second.
The company was happy with its SAP ERP deployment on earlier releases of Microsoft SQL Server and the Windows Server operating system. SQL Server 2008 R2 had provided great cost savings with its database compression—not only for the SAP ERP system, but for the whole SAP landscape where the databases of most of the SAP applications are completely page-level compressed.
Despite the growth of the SAP ERP system and the whole SAP application landscape, Microsoft still does not employ a database administrator (DBA) team to manage the SAP databases. The few necessary maintenance tasks are done by the SAP Support (Basis) team within Microsoft.
In January 2011, Microsoft IT deployed very early releases of SQL Server 2012 into a SAP ERP production-sized sandbox system. Over the course of the year, Microsoft tested various prereleases of SQL Server 2012 focusing on new features like AlwaysOn. The test phase finally ended with the deployment of an early beta release of SQL Server 2012 into the production SAP ERP system in November 2011. As expected, SQL Server 2012 proved that it was enterprise-ready even in its early beta release. It also opened up new possibilities for meeting and improving existing high-availability and disaster-recovery requirements.
“Our SAP ERP serves as the backbone of Microsoft’s business,” says Hans Reutter, Sr. Principal OE System Manager at Microsoft. “We never would upgrade this system to a new database release if we weren’t confident. SQL Server 2012 did meet expectations in all of our testing and proved enterprise-ready even in beta state.”
"SAP ERP serves as the backbone of Microsoft’s business. We never would upgrade this system to a new database release if we weren’t confident. SQL Server 2012 did meet expectations and proved enterprise-ready even in beta state.” Hans Reutter Sr. Principal OE System Manager Microsoft
"SAP ERP serves as the backbone of Microsoft’s business. We never would upgrade this system to a new database release if we weren’t confident. SQL Server 2012 did meet expectations and proved enterprise-ready even in beta state.”
Hans Reutter Sr. Principal OE System Manager Microsoft
When Microsoft upgraded its SAP ERP environment to SQL Server 2012 Enterprise, the SAP deployment had a three-tier architecture that includes:
Presentation Tier. The presentation tier includes the SAP graphical user interface, which is used by some 4,000 information workers. Meanwhile, all Microsoft employees are indirectly using the SAP applications through intranet applications for all kinds of tasks, including expense reporting, procurement, and employee self-services.
Application Tier. The application tier includes 11 servers running SAP application instances. These servers are running Windows Server 2008 R2 Enterprise and are a mix of HP ProLiant DL580 G5 and HP ProLiant DL580 G7 server computers with 64–128 GB of memory. Each of the servers is powerful enough to run between two and four SAP application instances. Instead of running very large SAP instances, the Microsoft SAP Support team prefers to have several smaller SAP instances on one of those powerful industry-standard servers.
Database Tier. The 5.8-terabyte SAP ERP database is hosted on the beta edition of SQL Server 2012 Enterprise, running on Windows Server 2008 R2 Enterprise. Thanks to SQL Server database compression, the database grows only about 120 GB a month. The full database is hosted on a single HP ProLiant DL580 G7 server with 4-socket 8-core processors and 256 GB of RAM. It is connected using fiber optics to an EMC CX3-80 SAN disk storage array.
To meet the high-availability and disaster-recovery requirements, the SAP ERP database is replicated locally and into the disaster-recovery site using SQL Server 2012 AlwaysOn technology. The second and third database servers and storage arrays are an exact copy of the primary server and storage array to assure a failover without degrading expected performance.
The SAP deployment took advantage of many features of earlier SQL Server releases including:
Data Compression. Using page compression for all data and indexes reduced the data volume in the database dramatically. The experience showed a compression factor of 3–5, especially for some of the larger tables without affecting the performance. On the contrary, some batch jobs actually ran faster. All in all, the disk space consumption throughout the whole SAP landscape could be drastically reduced using SQL Server database compression. This resulted in annual savings of hundreds of thousands of dollars in storage costs.
Backup Compression. Backup compression as first deployed with SQL Server 2008 reduced investments into infrastructure run times, and disk-space consumption for full database backups dramatically. Thanks to backup compression, a full database backup of 5.8 terabytes is executed in around 2:45 hours and results in a volume of about 1.6 terabytes on disk.
Database Mirroring. Database Mirroring has been used since 2005 to provide local high availability. Not only did the availability of the SAP ERP system improve against unplanned issues, but after seven years of using Database Mirroring, the SAP ERP system was available for planned maintenance more often. Planned maintenance includes upgrading platform software such as Windows and SQL Server, exchanging storage or servers, and moving servers or storage within the data center to new locations.
With the upcoming SQL Server 2012 release, the Microsoft SAP Support team was especially interested in leveraging the new AlwaysOn technology. The requirement in terms of Recovery Time Objective and Recovery Point Objective for a local failover or a disaster-recovery failover became more and more stringent as the SAP ERP system ran more and more business processes. What the Microsoft SAP Support team also looked for was one technology they could leverage for both local failover and disaster failover. To this point, SQL Server Database Mirroring has been used for local failover and Log Shipping has been used for disaster recovery.
"One reason we can achieve 99.995 percent uptime, even when counting scheduled planned platform downtime, is that we used SQL Server Database Mirroring in the past and AlwaysOn today." Eric Holling, Principal Service Design Engineer, Microsoft SAP Support Team
"One reason we can achieve 99.995 percent uptime, even when counting scheduled planned platform downtime, is that we used SQL Server Database Mirroring in the past and AlwaysOn today."
Eric Holling, Principal Service Design Engineer, Microsoft SAP Support Team
Upgrading to SQL Server 2012 Enterprise gave Microsoft one single functionality—AlwaysOn— that could be applied to both local high availability and disaster recovery. The solution also improved auditing capabilities and online table maintenance; offered support for Windows Core; eliminated the need for database administrators; and gained 99.995 percent availability.
Like Database Mirroring, AlwaysOn technology continuously sends a database's transaction log information from the primary database to a secondary replica. The originating database has the role of a primary, and the receiving database has the role of a secondary. As the primary server writes the primary database's log records to disk, it simultaneously sends that block of log records to the secondary instances. In contrast to Database Mirroring, which was restricted to one secondary replica of the primary database, AlwaysOn supports up to four secondary replicas with mixed-synchronization modes. This opens the opportunity to place one secondary replica within the same data center to cover local high-availability requirements. An additional secondary was placed in the disaster-recovery site, replacing the Log-Shipping functionality used with earlier SQL Server releases to cover the disaster-recovery needs. As the changes to the primary database of the SAP ERP system get replicated in a synchronous manner to the local secondary, the remote secondary in the DR site is supplied with these changes asynchronously. Within the same data center, the primary and secondary can automatically failover between each other. Together with the synchronous replication, this creates a local high-availability configuration of the SAP ERP database that will not lose any committed transactions. Replacing Log Shipping to the DR site with AlwaysOn technology also reduces the Recovery Point Objective in case of a manual failover to the DR site from 2–3 minutes to a few seconds.
AlwaysOn also resolves another problem the Microsoft SAP Support team often encountered. In order to refresh their SAP ERP test system with the most recent data from the production database, they needed to copy the 1.6-terabyte backup over to the DR site that also hosts the SAP ERP test system. This process could take many hours and required monitoring. With AlwaysOn, backups can be taken from the secondary replicas. Having one replica in the DR site, the team can take backups in the same site as the test system. Copying the 1.6-terabyte backup files over the network between data centers is no longer necessary.
“With AlwaysOn we finally have one solution that covers our high-availability and disaster-recovery needs,” says Elke Bregler, Principle Systems Architect in the Microsoft SAP Support Team. “Beyond that it saves us a lot of time by being able to perform full database backups in our disaster-recovery site from one of our secondary servers.”
AlwaysOn also simplifies monitoring of the complete high-availability and disaster-recovery scenarios dramatically. With the AlwaysOn dashboard, which is completely integrated into SQL Server 2012 Management Studio, configurations like the one Microsoft has can be monitored on one screen. Initial investigations can be started within this dashboard without even touching the secondary replicas first.
Although AlwaysOn—and in former releases Database Mirroring—is often thought of in terms of recovering from unexpected issues, Reutter says it also helps to minimize scheduled downtime when applying software updates and performing other maintenance. “We enjoy 99.995 percent and higher raw platform availability for our ERP database,” says Eric Holling, Principal Service Design Engineer, Microsoft SAP Support Team. “This figure doesn’t include application downtime due to applying SAP software updates, and it doesn’t include disaster-recovery exercises we carry out. But it does include downtime due to software upgrades to Windows Server and SQL Server and Security Patches. One reason we can continuously achieve 99.995 percent uptime, even when counting scheduled platform downtime, is that we used SQL Server Database Mirroring in the past and AlwaysOn today.” In addition, Holling notes that this 99.995 percent availability is achieved while running the database on beta releases of SQL Server 2012 quite frequently and for many months.
When applying software or hardware updates, the SAP Support team simply takes one of the secondary replicas of the SAP ERP database offline, applies software or hardware updates, and then brings it back online. Then they failover to the secondary, thus promoting it to the new primary, and then apply the software and hardware updates to the new secondary replica. Later they bring this secondary replica back online to be automatically synchronized with the primary replica again. The failover and promotion of the secondary to become the primary result in downtime, but very brief downtime of only a few seconds.
“Thanks to the server development over the last 10 years, we hardly see catastrophic hardware failures,” says Reutter. “You certainly need to take measures to prevent hardware or software issues causing severe interruptions to your business processes, no matter how rare they could be. But the big value we see from AlwaysOn in our day-to-day operations is that scheduled planned downtimes that used to require 15 minutes to hours are no longer needed. We just use AlwaysOn, and failing over from one server to the next is accomplished in a matter of seconds, with no data lost and no perceivable downtime.”
With SQL Server 2012, the SQL Server Auditing framework can track queries in the SAP databases that were previously done by administrators or by applications other than SAP. With the enhanced filter mechanisms, Security Administrators can define exactly which user should be tracked and which shouldn’t be. The Auditing framework also makes it possible to define exactly which tables are supposed to be tracked and which activities on the tables need to be tracked. In this way, the functionality is very flexible and can be used relatively inexpensively to fulfill various compliancy requests.
Reutter notes that Online Table/Index maintenance, introduced with SQL Server 2005, has also reduced the need for scheduled downtime, because it provides the ability to create, rebuild, or drop an index online. With the online index option, concurrent modifications can be made (insert/update/delete) to the underlying table while an index is being created or rebuilt. SQL Server 2012 added various improvements to the Online table maintenance that now make it possible to maintain all tables in SAP databases online.
“The Online Table Maintenance feature allows us to create indexes or reorganize data stored in SQL Server, without the need to take the system offline,” says Holling. “This is crucial for maintaining high availability and uptime with a system that's as mission-critical as SAP ERP. We are able to create new indexes against large and frequently used tables like EDI [Electronic Data Interchange] tables during online operation, which is really a great benefit.”
"With AlwaysOn we finally have ONE solution which covers our HA and DR needs. Beyond that it saves us a lot of time by being able to perform full database backups in our DR site from one of our secondary servers." Elke Bregler Principle System Architect, Microsoft SAP Support Team
"With AlwaysOn we finally have ONE solution which covers our HA and DR needs. Beyond that it saves us a lot of time by being able to perform full database backups in our DR site from one of our secondary servers."
Elke Bregler Principle System Architect, Microsoft SAP Support Team
With SQL Server 2012, installation on the Windows Core editions is possible. Running Windows Core instead of a fully equipped Windows with Internet Explorer and a lot of other components requires dramatically fewer reboots. Tests with Windows Server 2008 R2 showed that systems running on Windows Core or Windows Server 2008 R2 were considerably less affected by patches and reboots. Fewer reboots are especially important for mission-critical systems like DBMS servers underneath SAP applications. Because SQL Server 2012 can be installed and run on Windows Core, platform availability will increase. Therefore, the Microsoft SAP Support team, in conjunction with several teams in the Microsoft data centers, started planning on moving critical components like DBMS servers in their SAP landscape to Windows Core deployments.
Microsoft has been using SAP software for many years—and not just SAP ERP. At the moment, Microsoft leverages the following SAP applications: SAP ERP, SAP BW, SAP Supply Chain Management, SAP E-Recruiting, SAP Global Trade Services, and SAP Convergent Charging. Even with all these SAP applications having separate databases of many hundreds of gigabytes, or even terabytes, in size and volume, Microsoft never employed specific DBAs to administrate these large SQL Server 2012 and SQL Server 2008 R2 databases. It was decided not to invest in specialized database administrators for SAP-related databases for several reasons:
It became apparent very early on that SQL Server did most tuning automatically. With early SQL Server releases, an administrator never had to tune cache and memory areas. SQL Server did and still does this based on load and demand in a manner optimized for SAP applications.
SQL Server also does not require steady tuning on the storage side. This is a result of how SAP leverages SQL Server and how SQL Server distributes data via proportional fill. With the way SQL Server distributes the data over different files of a SAP database, the workload against each of the files is the same. Hence, configuration and administration of storage for the SAP databases are getting simpler and more unified. Rebalancing of files on the storage backend is no longer necessary.
Tasks like updating table and index statistics are not necessary with SQL Server 2012 because the solution does this automatically. In SQL Server 2012, this functionality was enhanced, which resulted in the stability of query response time.
There is no need to interfere with the SQL Server allocation and de-allocation algorithms. SQL Server was designed to align optimally with the way the Windows Operating System interacts with storage. Therefore SQL Server does not offer the option to change the sizes of internal block or page sizes.
Database compression no longer needs to be adjusted and redone. Database compression is a one-time setting with SQL Server. Once a table is set to row or page compression, all data inserted is compressed automatically. There is no need to redo compression or worry whether a certain SAP table should be compressed or not. The default way of installing SAP on SQL Server is to use page compression for all tables with no exceptions. Hence there is no need for further optimization around compressing database content. SAP databases are always installed with the most powerful and efficient compression method.
SQL Server monitoring data is closely integrated into the SAP DBA Cockpit framework. All important SQL Server administration and monitoring views and methods were integrated into the SAP DBA Cockpit framework. As a result, it was possible to monitor for the usual SQL Server functionality and SQL Server performance using only one dashboard-like view. With the close integration between SAP software and SQL Server, usual DBA tasks like query performance tuning can be performed from within the SAP applications. This allows close correlation of performance indicators of the particular SAP application and SQL Server side-by-side within the SAP application, instead of using separate SQL Server and SAP tools.
The only tasks remaining besides monitoring and some volume planning around SQL Server underneath SAP is monitoring high-availability and disaster- recovery functionality via the AlwaysOn dashboard. DBAs for SAP applications need to understand how the software is integrated into the DBMS system. For efficient analysis, they also need to understand the SAP database performance-tuning transactions (such as tracing to investigate response time issues) within the SAP application. Therefore, Microsoft decided several years ago not to assign DBAs to work on the SAP databases, but rather to educate all SAP Basis Support team members on SQL Server basics.
Microsoft gained the data compression, backup compression, integrity of change tracking, and other benefits it sought by upgrading its 5.8-terabyte SAP ERP database to a beta release of SQL Server 2012 Enterprise running on Windows Server 2008 R2 Enterprise. The combination of Database Mirroring in earlier SQL Server releases and AlwaysOn in SQL Server 2012 complemented by the Online Indexing functionality has reduced the need for scheduled downtime, helping the team to enjoy at least 99.995 percent uptime for its platform.
Windows Server 2008 R2 Enterprise
Microsoft SQL Server 2012 Enterprise
EMC CX3-80 SAN disk storage array
HP ProLiant DL380 G7 server computers
HP ProLiant DL580 G5 server computers
HP ProLiant DL585 G7 server computers