Microsoft SQL Server Development Customer Advisory Team : Performance and Scalability

New Version of Partition Management Utility Available

stuarto — Tue, 24 Feb 2009 00:19:00 GMT

I’m happy to announce availability of an updated version of the SQL Server Partition Management tool – now offering full support for SQL Server 2008 as well as improvements in handling SQL Server 2005. The tool is available along with source code at http://www.codeplex.com/SQLPartitionMgmt. If you’re not already familiar with the tool, it provides a set of commands (at the Command Line or via Powershell) to create a staging table on-demand (including all appropriate indexes and constraints) based on a specific partitioned table and particular partition of interest.

By calling this executable, with parameters, from maintenance scripts or SSIS packages, DBAs can avoid having to ‘hard code’ table and index definition scripts for staging tables. The tool eliminates the challenge of keeping such scripts in synch with changes to partition tables’ columns or indexes. It also provides a fast, single-command shortcut for the operation of quickly deleting all data from a partition.

The latest version supports new features in SQL Server 2008 such as filtered indexes, new data types, and partition-aligned indexed views. Enjoy!

-- Stuart Ozer / SQLCAT

SQL Server Scale Out

kevincox — Thu, 12 Jun 2008 14:00:00 GMT

After a few recent conversations with CIO and CTO types, I was amazed that there is still the concept out there that SQL Server doesn’t scale well. So I sat down to write a lengthy blog to dispel that rumor. After doing some research, I found that there is so much already written and available on the web that I decided to gather all the links in one spot and make this blog a launching point for your own research and reading pleasure.

After reading these, I’m sure you will agree that the SQL Server relational engine will scale to the largest data requirements. In fact, I will go so far to state that SQL Server will take on the toughest database challenges in the world. If you don’t agree post a comment here and let me know your opinion.

If you are already using SQL Server and want to know more about the technical solutions on how to make your situation scale or perform better, go to http://sqlcat.com and read the variety of technical notes and white papers posted there. On this site, you will also find the links to the SQL Server Best Practices site in the upper right corner. Both these sites contain some of the best how-to guides around SQL Server.

If you only have a few minutes to read one or two articles and want to get an overview of most of the scalability options, read the one by Roger Wolter on Scaling out SQL Server 2005 or the one by Bob Beauchemin. These two overlap a little bit but each has a unique perspective on scale out. They will help give you a vision on which option would be right for your project.

Scalability Architecture choices (use for Envisioning and Architecture phases of project)

Scaling Out SQL Server 2005, Roger Wolter, http://msdn.microsoft.com/en-us/library/aa479364.aspx

Planning, Implementing, and Administering Scaleout Solutions with SQL Server 2005, http://www.microsoft.com/technet/prodtechnol/sql/2005/scaleout.mspx , Bob Beauchemin

Service-Oriented Data Architecture: App-Server:Lite?, David Campbell, http://research.microsoft.com/research/pubs/view.aspx?tr_id=983

Academic / Conceptual oriented papers

Data on the Outside vs. Data on the Inside. An Examination of the Impact of Service Oriented Architectures on Data, Pat Helland, http://msdn.microsoft.com/en-us/library/ms954587.aspx

Distributed Partitioned Views

SQL Server Megaservers: Scalability, Availability, Manageability, Jim Gray and Richard Waymire. (old article but still has relevant parts) http://www.microsoft.com/technet/prodtechnol/sql/2000/plan/ssmsam.mspx

Implementation and Internals (Use for Detail Design phases of project)

Scalability Internals http://www.microsoft.com/technet/prodtechnol/sql/2005/scaleout.mspx

Architecting Service Broker Applications, Roger Wolter, http://msdn2.microsoft.com/en-us/library/aa964144.aspx

Achieving Scalability and Availability with Peer-to-Peer Transactional Replication, Michael Blythe, http://www.microsoft.com/technet/prodtechnol/sql/2005/p2ptranrepl.mspx

Scalable shared databases are supported by SQL Server 2005, http://support.microsoft.com/default.aspx?scid=kb;en-us;910378

Anything on this site: http://www.sqlskills.com/whitepapers.asp

Peer-to-Peer Replication

Inside Microsoft.com: Managing SQL Server 2005 Peer-to-Peer Replication, David Lindquist http://www.microsoft.com/technet/technetmag/issues/2006/07/InsideMSFT/default.as px

Service Broker

Inside MSDN: Building the MSDN Content Aggregation System, John Mollman, http://msdn.microsoft.com/en-us/magazine/cc163586.aspx

Data dependent routing

MSN: http://www.microsoft.com/technet/prodtechnol/sql/2005/scddrtng.mspx

Comparison

SQL vs Oracle RAC http://www.microsoft.com/sql/techinfo/whitepapers/oraclerac.mspx Note: this is a 2-year old doc and some of the RAC problems mentioned in this article have been addressed by Oracle.

SQL Server is being used to build some of the largest database applications in the world. Evidence of some these large projects is documented on http://www.microsoft.com/casestudies. Mission critical systems in core banking, telecom, stock exchanges, retail and many other industries are using SQL Server to solve their tough business requirements.

Kevin

New Best Practices Whitepaper published: "Best Practices for Migrating Non-Unicode Data Types to Unicode"

Sanjay Mishra — Tue, 22 Jan 2008 02:18:00 GMT

The database system in a company that is growing internationally must support multilingual characters in tables with Unicode data types. Existing databases that support only non-Unicode information must be migrated from non-Unicode data type to Unicode data type. This paper summarizes best practices for migrating database systems from non-Unicode to Unicode data type.

http://download.microsoft.com/download/d/9/4/d948f981-926e-40fa-a026-5bfcf076d9b9/SQL_bestpract_MigrationToUnicode.docx

New Best Practices Articles Published - Analysis Services Many-to-Many Dimensions: Query Performance Optimization Techniques

carl.rabeler — Sat, 22 Dec 2007 02:09:00 GMT

Many-to-many dimension relationships in SQL Server 2005 Analysis Services (SSAS) enable you to easily model complex source schemas and provide great analytical capabilities. This capability frequently comes with a substantial cost in query performance due to the runtime join required by Analysis Services to resolve many-to-many queries. This best practices white paper discusses three many-to-many query performance optimization techniques, including how to implement them, and the performance testing results for each technique. It demonstrates that optimizing many-to-many relationships by compressing the common relationships between the many-to-many dimension and the data measure group, and then defining aggregations on both the data measure group and the intermediate measure group yields the best query performance. The results show dramatic improvement in the performance of many-to-many queries as the reduction in size of the intermediate measure group increases. Test results indicate that the greater the amount of compression, the greater the performance benefits—and that these benefits persist as additional fact data is added to the main fact table (and into the data measure group).

http://www.microsoft.com/downloads/details.aspx?FamilyID=3494E712-C90B-4A4E-AD45-01009C15C665&displaylang=en

New Best Practices Whitepaper Published: The Impact of Changing Collations and of Changing Data Types from Non-Unicode to Unicode

Sanjay Mishra — Fri, 07 Dec 2007 02:07:00 GMT

When a business grows internationally, its database system must support multilingual characters in tables. Businesses that previously needed to manage only non-Unicode information must now support database tables that include non-Unicode data types such as CHAR or VARCHAR. As the business grows internationally, it must alter its database system to support Unicode data types. It may also need databases that have different collations across several servers.

http://download.microsoft.com/download/d/9/4/d948f981-926e-40fa-a026-5bfcf076d9b9/SQL_bestpract_CollationChange.docx

Using ByAttribute or ByTable Processing Group Property with Analysis Services 2005

denny.lee — Sat, 20 Oct 2007 07:19:00 GMT

As noted within the Analysis Services 2005 Performance Guide, there are some niche situations where setting the Processing Group property to ByTable provides more optimal processing than the default value of ByAttribute.

In a customer scenario, we had discovered that they had two dimensions (each of which has >25 million members and 8-10 attributes) where the Processing Group property was set to ByTable. While the ByTable setting could theoretically have Analysis Services process faster (because it takes the entire set of dimension data and places it in memory), it didn’t in this case because it had taken approximately 80% of all available memory (approximately 25.6GB out of 32GB physical memory) to place just one dimension into memory. Due to the large size of the dimensions, there were also issues in the dimension processing completing in a timely manner. The setting ByTable is an optimization that bypasses normal checks and assumes that there is enough memory to process all attributes concurrently in memory. If this is not true, this may result in processing issues and/or errors. Therefore, it is important for you to monitor memory usage; especially if dimension size grows over time.

Note, we had investigated using the MaxParallel setting to limit concurrency. Often this is helpful during partition processing, but it was not helpful during dimension processing for our scenario. We feel it was not useful because the root cause is the large amount of memory consumed when using ByTable, and either it is only one dimension causing the problem, or the memory quota mechanism was sufficient to prevent doing two very large dimensions concurrently.

With the attribute set to ByAttribute, the maximum amount of memory Analysis Services had to take up during processing was 9GB (vs. 25.6GB on the same server). That is, the default setting in many cases will use less resources and process in a timely manner. More specifically for this scenario, the reason for the processing issues was because of the large size of the dimensions (# of members, # of attributes, etc.) and when using the ByTable setting, AS had tried to put the entire dimension into memory instead of delegating this process to the relational database (i.e. when using the ByAttribute setting).

Contributors: Denny Lee, Richard Tkachuk, Akshai Mirchandani, Eric Jacobsen

SSAS small/big query interaction

Eric Jacobsen — Sat, 20 Oct 2007 03:24:00 GMT

Default SSAS behavior can sometimes result in small queries being slowed down by concurrently running big queries.

Recently in working with a customer, we observed the most common queries when run by themselves took about 1 second and some ad-hoc queries against their 110 GB cube took about 3 minutes. When running both together, small query times slowed down to several minutes. This slowdown prevented the customer's project from being successfully deployed.

There is a configuration parameter that can alter the scheduling of the tasks to allow separate queries to run more independently. The SSAS version that includes the configuration parameter is described in KB article 922852. We worked with the customer to develop a table of configuration parameter values and the resulting query times. Some configuration values resulted in the small query running concurrently with big queries in about the same time as when run independently.

The scenario we are focused on is a big query that takes substantially more time than a small query, and when run concurrently the small query is observed to substantially slow down. The root cause is the big query spends time in one or more storage engine requests, reading from many partitions having tens or hundreds of gigabytes. Many tasks are queued to a thread pool to handle partitions and segments inside the partitions.

This scenario and its solution may apply to other customers. However the effectiveness of a proposed solution is based on the root cause of the slow down. if the root cause is different there might not be a benefit from the same solution.

The steps below can be used to determine similarity to the scenario being discussed. When working with CSS you may be guided to repeat the steps with a different configuration parameter.

Steps to reproduce scenario:
1. Start small query, wait for completion, record time.
2. Start big query, wait for completion, record time.
3. Restart server.
4. Start big query (but do not wait for completion).
5. Wait 5 seconds then start small query. This wait time might need to be adjusted to get consistent times, for example to 10 or 15 seconds. The effect depends on the nature of the big query.
6. Wait for queries to complete, record times.

Customers having this circumstance can work with Microsoft Customer Support Services (CSS) and mention CoordinatorQueryBalancingFactor. Please keep in mind server properties that are not public can only be changed after consulting with Microsoft Support.

Using unformatted cell values it may improve query performance

Nicholas Dritsas — Wed, 26 Sep 2007 04:40:00 GMT

If a report does not use formatted cell values, query time can be reduced. An example is a Reporting Services report that does its own formatting and therefore would not use the formatted values from SSAS. By returning only the value of the cell and not its formatted value in MDX, you can achieve better queries performance, sometimes between 5-20%, depending on the size of the cell set. The default properties include FORMAT_STRING and FORMATTED_VALUE, and can be omitted by specifying a list of properties. An example is to add "cell properties VALUE" at the end of every query.

select [Measures].[Internet Sales Amount]

on columns

from [adventure works]

cell properties value

New Best Practices Article

carl.rabeler — Wed, 12 Sep 2007 22:10:00 GMT

Scale-Out Querying with Analysis Services Using SAN Snapshots

SQL Server Best Practices Article: Identifying and Resolving MDX Query Performance Bottlenecks in SQL Server 2005 Analysis Services

carl.rabeler — Wed, 29 Aug 2007 10:06:00 GMT

Identifying and Resolving MDX Query Performance Bottlenecks in SQL Server 2005 Analysis Services

There are a variety of things that you can do to improve the performance of an individual MDX query. To begin, you must identify the source(s) of the performance bottlenecks in the execution of a poorly performing MDX query. To identify the source(s) of performance bottlenecks, you need to understand how to use the available Microsoft Windows and SQL Server 2005 Analysis Services troubleshooting tools to assist you with identifying bottlenecks and learn how to interpret the information revealed by these tools. This article provides the information about the available tools and demonstrates how to use the most common of these tools to identify and resolve MDX query performance bottlenecks with individual MDX queries.

Performance Analysis Tool - DMVStats

lingzhuz — Fri, 13 Jul 2007 07:38:24 GMT

Tom Davidson wrote:

I've been working with database applications for the last 25 years. A particular area of interest for me is SQL Server performance. As part of the Microsoft SQL Server Development Customer Advisory Team (SQLCAT) for the last 7 years, I've had an opportunity to observe what works (and more importantly what doesn't !) in the enterprise space. One of the key enterprise needs prior to SQL Server 2005, was an effective mechanism for troubleshooting and performance tuning for many common user scenarios. SQL Server Development along with SQLCAT & PSS collaborated on defining these common scenarios as well as a mechanism for troubleshooting and performance tuning that ultimately resulted in the SQL Server 2005 feature called Dynamic Management Views (DMVs). DMVs provide a consistent and transparent interface to view changing server states.

SQL Server 2005 SP2 does include a helpful Performance Dashboard comprised of a series of Reporting Services reports to identify some performance issues using query statistics such as execution counts, IOs, worker time, and query plans - courtesy of Microsoft's stellar PSS SQL escalation team - Keith Elmore, Bob Ward, Bob Dorr et al. (see Performance Dashboard for more info). While SQL Server 2005 provides all this great new DMV plumbing, there are a whole series of common user scenarios that are difficult to analyze and pinpoint. Examples include:

what happened yesterday or last week? e.g. historical forensics,
trending,
source of obscure or transient waits (*not included in query stats),
comparisons before and after application changes

Over the last year, I've been working on such a tool called DMVstats with some of my CAT colleagues. DMVstats collects performance oriented DMVs into a data warehouse, and provides a methodology called 'Waits' and 'Queues' to identify and track down performance issues. Drill-through analysis is provided by reporting services reports.
DMVstats can be downloaded at http://www.codeplex.com/sqldmvstats. Please comment on the DMVstats tool - your comments & suggestions - good or bad, all are welcomed!

SQL Server 2005 Transactional Replication: Benefit of using SubscriptionStreams for low bandwidth, high latency environments

Sanjay Mishra — Tue, 08 May 2007 00:00:00 GMT

Author: Bren Newman

Introduction

When using Microsoft SQL Server 2005 Transactional Replication, the Log Reader Agent reads the log, constructs SQL statements for published articles and posts it to the Distribution database; the Distribution Agent reads the Distribution database, delivers and applies the "package" to the Subscriber. Often the Publisher, Distributor and Subscribers are on fast networks resulting in very low total delivery latencies. However on slower networks such as in a geographically distributed environment, the connection speeds are often of low bandwidth with high network latencies - this obviously has a negative effect on total delivery time. Network latency (easily seen when using the ping command) has a substantial negative performance impact and using the new -SubscriptionStreams can provide drastic performance increases.

-SubscriptionStreams NN

By default (and all previous editions of Microsoft SQL Server) the Distribution Agent uses as single stream to deliver a transaction from the distribution database to the subscriber database. In Microsoft SQL Server 2005 this default behavior can be overridden by specifying the -SubscriptionStreams NN parameter, where NN can be 0 (for non-SQL Server Subscribers or Peer-to-Peer Transactional subscriptions) to 64; where 64 would be a pretty insane number of threads especially if there are multiple Distribution Agents concurrently executing on the same server. The parameter refers to the number of connections allowed per Distribution Agent used to apply batches of changes in parallel to a Subscriber. It's important to note that even though transactions are parallelized, transactional consistency is maintained. This is achieved by cleverly hashing and "bucketizing" the primary key and re-assembly before committal at the subscriber. However, as noted in BOL (ms-help://MS.SQLCC.v9/MS.SQLSVR.v9.en/rpldata9/html/67084a67-43ff-4065-987a-3b16d1841565.htm), if one of the connections fails to execute or commit, all connections will abort the current batch, and the Distribution Agent will use a single stream to retry the failed batches. Before this retry phase completes, there can be temporary transactional inconsistencies at the Subscriber. After the failed batches are successfully committed, the Subscriber is brought back to a state of transactional consistency.

The requirement

A very large industry-leading online business required data to be replicated from busy OLTP servers via the internet to geographically distributed offices around the world with low latency and be able to recover/catch-up from loss of connection for up to 24 hours. The "last-mile" local connection speeds varied but were generally of poor quality (i.e. high latency) and low speed/bandwidth. The average office (i.e. subscriber) connected at around 512kbps with average latencies of between 200-600 milliseconds. The Publisher (i.e. OLTP server) and remote Distributor are located on a fast local network and the Distributor server is a dual-core four socket processor.

The test

The OLTP publisher was "primed" with hundreds of transactions each consisting of approximately 5 to 10 commands per transaction. The Logreader Agent then moved the replicated transactions into the Distribution database. The Distribution Agent was then started and the duration and throughput measured. The network connection between the Distributor and the subscriber was throttled using Network Throttling software (which interfaced directly with network card). Network bandwidth of 512kbps and 128kbps with latencies of 300 milliseconds was tested.

Test results/findings

On a throttled network with 512kbps bandwidth, 300 millisecond latency and a default of only 1 distribution stream; a throughput of 6.86 commands per second was distributed to the subscriber. When increasing the number of streams to 8; a throughput of 40.34 commands per second was attained - 488% improvement. And when further increasing the number of streams to 16; a throughput of 56.11 commands per second was attained - 718% overall improvement and 39% improvement over 8 streams.

When the network bandwidth was reduced to only 128kps but the network latency kept at 300 milliseconds; 1 distribution stream delivered 5.9 commands per second, 8 distribution streams delivered 31.54 and 16 distribution streams delivered 28.01 commands per second. Chart number 1 shows these results.

Although it's important to notice the hugely positive throughput benefit of using SubscriptionStreams, it's important to note the impact of reduced bandwidth versus increased network latency. The results indicate that network latency has greater negative effect than reduced bandwidth - due to the longer response/acknowledge time for each transaction to be delivered across the network. It's equally important to note that for this test, with very low bandwidth (with high network latency), 16 subscription streams did not improve the delivery rate (in fact, there was a slight decrease). This indicates that bandwidth utilization has been exceeded and there is a negative cost adding more managed parallel threads.

As the requirement included the ability to recover/catch-up from a loss of connection for up to 24 hours, Chart number 2 shows the effect of the positive performance improvement measured in hours. Over a slow network, without Microsoft SQL Server 2005 Subscription Streams, it would have taken up to 11 hours to catch-up whereas with the feature less than 2 hours is possible. This assumes 24 hours worth of transactions or 240,000 commands waiting to be distributed due to network connection outage (or other scenarios).

Conclusion

The SubscriptionStreams feature in SQL Server 2005 Transactional Replication can have a highly positive effect on performance, especially over a slower networks; however it must be cautioned that the more threads/connections are opened and if an "error" occurs during reassembly and or committal of the transactions at the subscriber, the number of streams are reduced to 1 and at a later stage increased back to the configured value. The constant toggling between the two states can be costly. It is highly recommended that the optimal number of streams must be carefully tested for each application and environment before rollout.

Even on a high bandwidth network with low network latency (e.g. 1gigabit and 0 millisecond latency), configuring the Distribution Agent to use SubscriptionStreams can provide significant performance benefits - as illustrated in Chart 3. Clearly this is a feature worth testing and possibly deploying.

How to warm up the Analysis Services data cache using Create Cache statement?

Nicholas Dritsas — Thu, 26 Apr 2007 12:55:00 GMT

Goal

This document describes how to build Create Cache commands. Create Cache for Analysis Services (AS) was introduced in SP2 of SQL Server 2005. It can be used to make one or more queries run faster by populating the OLAP storage engine cache first.

Some customers have found certain queries benefit other later queries. For example, ascmd.exe could be used every hour to execute all queries in a directory keeping the cache ready for subsequent user queries. The other approach, which has been used, is to create a cache query for each user query. This is feasible if the MDX query is part of a report, then one simply adds another query that has the side effect of populating the cache, thereby speeding up the next query.

The root of the problem is that during a query, the AS Server does only local optimizations. Calculations, mixed granularities, and other more complex MDX statements can result in a chatty communication between the FE (Formula Engine) and the SE (Storage engine). In AS2000 and earlier, this was also a network round trip.

By issuing a cache statement, we can populate the cache with one or more subcubes that cover the regions of cube space that the query will actually need. We often find approximately the same time taken for each subcube query, so the effect can be dramatic overall.

With this methodology, the collection of MDX queries will appear as first executing inside the storage engine, and second inside the formula engine. In addition to reducing overall time, this can make it easier to predict the effect of multi-user load testing, because the first part uses 100% of all CPUs, and the second part uses 100% of one CPU.

Summary Steps

It is an iterative process. I would describe the identification of potential scenarios where create cache would help:

Run profiler
Run the query looking for Non-cache SE queries (query subcube filtered by subevent = 2)
Look at the total time of the query vis-à-vis of the sum of the times of the non-cache SE queries.
If:

They are pretty close and
There are many non-cache SE queries within the same range of time (not just 1 or 2 time consuming)

then create cache might help.

There could be variations, but here are one set of steps that have been used successfully:

1. Extract all MDX queries as separate files.

2. Add a Create Cache statement to correspond for every MDX query file.

3. Run Clear Cache, Create Cache, then the user query.

4. Verify that the Create Cache is effective and improve, if necessary.

5. Work on next query

6. When done with all queries, combine the Create Cache into 1 or more Create Cache queries.

7. Verify the combined Create Cache.

Detailed Description

Below are details about each of the steps.

1. Extract MDX queries as separate files

Placing the queries in separate MDX files, it can be faster to work one by one and verify if each query is handled correctly by Create Cache before moving on.

1. Start a trace.

2. Run the report.

3. Stop the trace.

4. Extract the queries. One way is with SQLProfiler, /File /Export /Extract SQLServer Analysis Serverices Events / Extract All Queries. This creates a text file, with each query on a separate line. When queries have multiple lines this might be confusing, but is easy to add an extra line or otherwise edit. Either copy each query to a separate file, or, for every query that is being worked on, comment out the other queries.

2. Add Create Cache for every MDX query

Below is an example of a Create Cache statement.

create cache

for [MyCube]

as (

{ [USA].[Oregon], [USA].[Colorado], [USA].[Florida], [USA].[Washington] }

* { [Measures].[mybasemeasure] }

* { [2006].children, parallelperiod( [Time].[Year], 1, [2006].[Q1].[Jan] ), YTD( parallelperiod( [Time].[Year], 1, [2006].[Q1].[Jan] ) ) }

* { [Products].[Shoes].children }

)

It should be apparent that it is basically a crossjoin of each dimension member that is specified in the query. Note that set expressions are allowed.

2.a. Add Specified Members

First add all members specified in the query. NOTE: Create Cache covers static analysis rather than dynamic. MDX with dynamic members will not benefit from this approach.

2.b. Add Calculated Members and Definitions

During execution, the Formula Engine can issue a subcube query which includes calculated members.

For example, if the cube has a calculated member:

[MyDim].[Calc123] as [MyDim].[A] + [MyDim].[B]

one should include the following members in Create Cache:

[MyDim].[A], [MyDim].[B] }

2.c. Account for Custom Rollups

During execution, there may be custom rollups that affect the calculation. For example, a custom rollup may involve QTD() or YTD(). This might be hard to detect since the actual calculations are stored in the relational database. Custom rollups can be viewed from BI Development Studio. The dimension must be processed (otherwise the calculations still only exist on the relational database).

1. Go to Solution Explorer, double-click on the dimension you want to examine.

2. Click on the Browser tab.

3. Find the icon for Member Properties, click on it. Select "Custom Rollup", "Custom Rollup Properties".

4. It will display the Custom Rollup (formula) and other properties (such as solve order) for each member.

For example, if calculations include YTD or PeriodsToDate, the member list should account for that. So, if the query includes:

[Date].[2006].[June],

ParallelPeriod( [Date].[2006].[June], [Date].[Year], -1 )

YTD( [Date].[2006].[June] )

the only calculations resulting in output are [Date].[2005].[June], [Date].[2006].[Jan] ..[Date].[2006].[June]. However, because of indirect relationships, you should also add the member set:

YTD( ParallelPeriod( [Date].[2006].[June], [Date].[Year], -1 )

3. Run Clear Cache, Create Cache, User Query

Now, it is time to run the Clear Cache statement and examine its effect. Note that the Cube ID can be specified or left blank to clear the entire database cache entries.

<DatabaseID>FoodCmp</DatabaseID>

<CubeID>SalesCube</CubeID>

</Object>

</ClearCache>

</Batch>

3.a. Connection string parameters

We recommend these parameters to be added at the connection string.

Provider=msolap.3;Datasource=MyServer;Initial Catalog=MyDatabase;Timeout=300;Disable Prefetch Facts=true;Cache Ratio=1

Provider: The name of the OLE DB provider, msolap.

Datasource: The name of the server. Might be named instance like MyServer\MyInstance.

Initial Catalog: The name of the database to use.

Timeout: Optional number of seconds for command timeout. This can help to avoid very long runs until the queries are optimized to run faster.

Disable Prefetch Facts: Optional new parameter to disable the Formula Engine heuristic that sends queries for possibly more data than it is requested.

Cache Ratio: Optional parameter to control the conversion from a specified set of members to the subcube query sent to the SE Storage Engine. Use 1 to send only the specified list.

3.b. Ways to Execute Queries

Because the connection string needs to be modified, SQL Server Management Studio cannot be used. There are several alternatives:

1. One could write a small program in C#.

2. ascmd.exe can execute queries and output time. The ascmd utility is part of the Yukon Samples kit – located here:

http://www.microsoft.com/downloads/details.aspx?FamilyID=e719ecf7-9f46-4312-af89-6ad8702e4e6e&DisplayLang=en

Then look in the C:\Program Files\Microsoft SQL Server\90\Samples\Analysis Services\Administrator\ascmd folder.

3. Use the mdx sample app from AS2000.

4. Verify Create Cache is Effective

The goal of using Create Cache is to isolate the storage engine and formula engine work and prevent the user query from executing subcube commands. The SE is multi threaded when the FE is single threaded. Therefore, one can just use Task Manager to verify if the SE is running.

During a good run, initially all the CPUs are at 100% since the SE is busy with the Create Cache command. Then, only one CPU reaches 100% because the FE is busy with the formulas and preparing the result set.

Task Manager is the general way to verify, but to be sure, create a trace and look for event Query Subcube, subevent 2 Non Cache. The cache subcube queries are fast and generally not an issue for AS. (In fact it shows how effective it is to create a cache first.)

4.a. Improve, if necessary

If the user query results in Task Manager CPU spikes (caused by subcube queries), one only needs to determine which members from the main query are not present in the Create Cache statement.

It might be helpful to look at trace information for Query Subcube Verbose. It might be that by comparing the ones for the Create Cache with the ones for the user query, some differences can be seen, and that could help discover which members should be added to the Create Cache query.

Another approach is to simplify the query until the problem does not happen. Try to use binary search to find the critical part of the user query that causes the extra subcubes.

4.b. Eliminate All Subcubes

This general technique works well when one can eliminate all subcubes. However, it might not be possible to know the set of members in advance (static analysis). If this is not true, for instance for complex calculations, then the Create Cache technique might not be as beneficial. Adding members that are not used can be cheap in some cases, for example to add one more product in an already large specified set. Or it might be expensive, for example to include another large set of data that otherwise would avoid entire partitions. One will need to experiment and measure the result, and because sometimes it will be a win and sometimes a loss, we recommend considering several user parameters.

5. Work on Next Query

6. After working with all the main queries, combine the Create Cache into 1 or More Create Cache statements

The best number of Create Cache queries depends on their structure (which dimension members involved, which measure group) and member sets.

1. Different measure groups should be separated unless the dimension members are consistent. Then, it is just a matter of convenience to combine, since under the covers the measure groups are physically separate.

2. If one query is covered or almost covered by another one, they should be combined.

3. If queries are disjoint or mostly disjoint, keep separate.

We have not experimented greatly, but here is one suggestion. Execute both scenarios, as separate and combined. Measure execution time and size of resulting DataCache (from perfmon counter). Use single combined Create Cache if the combination reduces time to 67% of separate commands and consumes space less than 3 times as much as separate.

7. Verify the combined Create Cache

Execute all queries, in this pattern:

1. Clear Cache

2. Create Cache (possibly many queries)

3. User queries

Verify the set of queries in the same way as each single query.

Contributors:

Eric Jacobsen

Cristian Petculescu

SSAS Partition Slicing

Eric Jacobsen — Tue, 06 Mar 2007 01:10:00 GMT

Summary:
SSAS uses partitions to contain cube data. Small cubes might use only one, but for non trivial cubes, cube designers will create partitions based on ease of managing data and to split groups of data. This document discusses how the server uses "slices" to examine partitions at query time. The slices are stored as a range of internal surrogate keys, and can be thought of as a very high level index.

Using more attributes in the partition definition can potentially improve performance by the ability to exclude partitions. To benefit user queries successfully requires some consideration of the internal way partition slices are used. Note that even though slices might not exclude partitons, we still expect the map indexes to be efficient at retrieving only neccessary data.

Slice description:
By "slice" we mean one or more members that limit the multidimensional space. Queries may have a "slice" as a result of including a set of members on an axis, or by placing members or a set in the where clause. Query slices are always a finite set of members. The formula engine may decompose MDX queries into many storage engine queries, which have a simple specification of granularity of each attribute, and a set of slices that limit the contents.

Cube partitions each have a slice for each attribute. This slice is either set by the administrator as part of the partition definition, or for MOLAP partitions is created during creation of the bitmap index for the partition. There is one slice per attribute in the partition. (Similarly there is one bitmap index per attribute in the partition.) Because the server automatically discovers the slice it is sometimes referred to as the "auto slice".

The partition slice value is stored in a file in the partition directory, with the name "Info.*.xml". The information stored is the minimum and maxiumum member IDs that are present in the partition, for each attribute. The member IDs are internal to the olap server, and are basically surrogate keys, created as a contiguous set of 32-bit integers starting with 1. This can have an impact when considering the min/max range way that partitions are created. It might be beneficial to change the SQL query for the dimension attributes so that the order in which attributes are returned matches the partitioning slicing scheme. Then the alignment of the internal surrogate keys helps the server at query time eliminate some partitions from being considered. See below the section Design Example.

The slice is created during index creation. Index creation is controlled by the number of rows, specified in the config file as <IndexBuildThreshold>, with a default of 4096. Partitions with fewer rows will not have indexes built -- and therefore will not have an "auto slice".

Optimal Number and Size of Partitions

There is not necessarily an optimal number or size of partitions, but some general guidelines exist. In Analysis Services 2005 Performance Guide (see link below), we suggest at most about 2000 partitions, and maximum of about 2 GB or 10-15 million rows in size. Some sites successfully use much larger partitions. For number of partitions, too few can miss opportunities to use slicing or can cause larger time windows for processing. Too many partitions might take more work to manage, cause some slowdowns of operations during processing (improved after SP2 was released), and can slow service startup time due to many more files on disk. Too small partitions might not offer any benefit. Too large partitions might increase the time window for processing past what is available.

Query Time Behavior

During query time all partitions are scanned for possible inclusion, by checking the query slices and partition slices for intersection. This is sequential, but is fast because it involves only integer range comparisons. Partitions that are included will be queried in parallel, and partitions that are excluded are not touched further.

There are SQLProfiler trace events for each partition that is used. Which trace event depends on whether the fact table or an aggregation, and each partition could have a separate aggregation definition.

Troubleshooting Information

It is important for very large cubes that a partition is not touched unless the query depends upon it. The information generally used to troubleshoot is:

Trace event Query Subcube Verbose. Check to see which attributes have slices.
Info.*.xml file, located in the partition. Make sure slices are defined for the attributes used in the query.
Trace events, can be used to check which partitions are being used. See below.

In SSAS2005 SP2 and earlier, the following behaviors may have a negative impact on the number of partitions touched.

Aggregations having fewer than <IndexBuildThreshold> rows are not excluded based on slice, even if a slice range exists for the fact table data. One possible work around is to lower the value of the <IndexBuildThreshold> configuration parameter in the server configuration file. However, this is not recommended as it may result in a large number of additional indexes, slowing processing performance.
Queries using "OR" slices do not use partition slices. The OR slice can be identified by a "+" in the trace event Query Subcube Verbose.

Both of these issues are expected to be addressed in a future Analysis Services release.

Trace Events

These are the relevant trace events to examine.

Query Subcube Verbose. This trace event is emitted after the internal storage engine subcube query has completed. The duration shows the total time for creating the datacache. Note that a query will only have one occurring at a time, and many partitions can be read from concurrently. So the start and end times will not overlap other Query Subcube Verbose events. Here is an example part of the output.

D:3 (Time) [1 8 52 * 0 0 0 0 0] => ((All)):[All] (Season):[Winter 2007] (Period):[JANUARY_2007] (Week):* (Day):0 (Dimdate):0 (Year):0 (Half):0 (Qtr):0

The numbers in square brackets [1 8 52 * 0 ...] have these meanings:

Single number - The internal surrogate key ID for the member participating in the slice. Note that the first element is often the All level, which by definition only has 1 member, and therefore its ID will always be 1. The member name is displayed.

'*' - The answer must have granularity for this attribute, and all members are used. There is no slice on this member.

0 - The answer should not include granularity for this attribute.

'+' - A slice is defined with more than 1 member. This is sometimes called an "OR" slice, because the equivalent SQL query would be of the form: where city = 'Seattle' or city = 'Redmond'. The list of members is not displayed. For simple MDX queries it is sometimes possible to infer the list.

Query Subcube. This trace event is not as readable for a human as Query Subcube Verbose. The format is a bitmap of attribute granularities, with dimensions delimited by a comma (",").

The same format of information is emitted by the Get Data From Aggregation trace event, and is also inserted into the QueryLog relational database table, if enabled in the configuration file. You can cut and paste this string directly into the AggManager sample to help create aggregations that match your queries, without or in addition to using the QueryLog relational table. (We wish to point out the format can be transferred -- not to suggest creating aggregations by just cutting and pasting. Designing the set of of aggregations is beyond the scope of this document.)

TextData from Query Subcube event : 00000000,000000000000000000000000,01110000

In this example, there are 3 dimensions, and the query only has granularity on attributes in the 3rd dimension. (Which for this example happened to be the Time dimension.)

Get Data From Aggregation. This trace event is emitted when reading from a partition, in an aggregation (not from the lowest level fact data). It is not emitted if the partition can be excluded based on the partition's "auto slice".

Note that the start and end time will typically overlap the start end end time of other events, because storage engine reads partition data in parallel, and the time will be contained within the time range for Query Subcube and Query Subcube Verbose.

The ObjectPath trace column tells what partition is being read. Note that Query Subcube and Query Subcube Verbose also have ObjectPath, but are missing the partition information since the query represents the union of all its partitions. The format is as follows. Note that for named instances the server name might be of the form MachineName\ServerInstanceName.

ServerName . DatabaseID . CubeID . MeasureGroupID . PartitionName

The TextData column contains the name of the aggregation used and the list of attribute granularity for the aggregation used. It will always "cover" (be larger than) the granularity of the Query Subcube. Here is an example:

TextData from Query Subcube event : 00000000,000000000000000000000000,01110000

TextData from Get Data From Aggregation event: Aggregation 1 00000000,000000000000000000000000,01110111

Progress Report Begin, Progress Report End These trace events are emitted when reading from a partition, in the fact data (not from an aggregation). It is not emitted if the partition can be excluded based on the partition's "auto slice".

The ObjectName trace column tells what partition is being read. The TextData column also tells the same information. Here is an example:

TextData from Progress Report Begin event : Started reading data from the 'MyPartitionName' partition.

Example of Partition Slice Information

The exact MDX query is unimportant, so we will just look at the Query Subcube Verbose and some partition information.

Query Subcube Verbose, from SQLProfiler trace:

Dimension 0 [Location] (0 0 0 0 0 0 0 0) [Dimlocation]:0 [Chain]:0 [Division]:0 [Region]:0 [Area]:0 [Store]:0 [Location Type]:0 [Grouped Stores Ind]:0
Dimension 1 [Product] (0 2 2 14 137 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0) [Dimproduct]:0 [Company]:[C SAMPLE] [Brand]:[B 1 SAMPLE] [Group]:[G 105 SAMPLE] [Dept]:[D 1125 SAMPLE] [Category]:0 [Range]:0 [Style]:0 [Colour]:0 [Size 1]:0 [Size 2]:0 [Size]:0 [Season Ind]:0 [Replenishment]:0 [Clearance]:0 [RSP]:0 [Primary Supplier]:0 [Supplier Color]:0 [SKU]:0 [Size Indicator1]:0 [VPN]:0 [Sku Description]:0 [Latest Cost]:0 [Original RSP]:0
Dimension 2 [Time] (0 * * * + 0 0 0) [Dimdate]:0 [Week]:* [Period]:* [Season]:* [Day]:+ [Year]:0 [Qtr]:0 [Half]:0

The "0" means no granularity for that attribute.

The "*" means include granularity for that attribute, with all members.

The "+" means include granularity for that attribute, with a set of members.

A number means include granularity for that attribute, with one specific member. The number is the internal surrogate key.

This subcube query included a slice on a set of days. Though the set members are not included in the trace event, for the actual query the set of [Time].[Day] for this example was { 1465, 1466, ... 1492 }.

So we can expect for this query, that partitions slicing by [Product] or [Time] would be effective at reducing query time by using slice information as a high-level index and reducing the work for the query.

SQLProfiler Trace information:

For this example, aggregations were able to answer the subcube query. So the relevant trace events are Progress Report Begin, and Progress Report End. Here are the relevant columns for one trace event. (Note that in SQLProfiler, this is displayed horizontally as one row.)

EventClass: Progress Report Begin

EventSubclass: 14 - Query

TextData: Started reading data from the 'Aggregation 9' aggregation.

DatabaseName: Sample

ObjectName: Aggregation 9

ObjectID: Aggregation 9

ObjectPath: MyMachine.Sample.Sample SMDB.Fact Sales SKU Store Day 0.Sales_W200639.Aggregation 9

Note that the partition name is part of the ObjectPath, a trace column that is disabled by default.

Here is a subset of info.*.xml for a customer partition year 2005 week 36. We can see that if the query slice contains any of the Product or Time properties, partitions should be excluded. Note that some ranges are narrow (few members), and some ranges are broad. For example, the range for [Product].[Colour] is 2...736, which is not surprising since the product sold in this week probably contains every color. If there was a goal of partitioning also by color or some other product attribute, that would help queries that sliced by that attribute.

For the example query subcube, we expect it to be excluded based on [Time].[Day]. For this example, it actually was excluded due to the set of days not in the inclusive range 1101...1107 for the property Day.

      <MapDataIndex>
        <DimensionID>Product</DimensionID>
        <PropertyID>ID Company</PropertyID>
        <m_MinIndex>2</m_MinIndex>
        <m_MaxIndex>2</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Product</DimensionID>
        <PropertyID>ID Brand</PropertyID>
        <m_MinIndex>2</m_MinIndex>
        <m_MaxIndex>2</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Product</DimensionID>
        <PropertyID>ID Group</PropertyID>
        <m_MinIndex>2</m_MinIndex>
        <m_MaxIndex>21</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Product</DimensionID>
        <PropertyID>ID Style</PropertyID>
        <m_MinIndex>22</m_MinIndex>
        <m_MaxIndex>325486</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Product</DimensionID>
        <PropertyID>ID SKU</PropertyID>
        <m_MinIndex>122</m_MinIndex>
        <m_MaxIndex>1307861</m_MaxIndex>
      </MapDataIndex>
        <DimensionID>Product</DimensionID>
        <PropertyID>Colour</PropertyID>
        <m_MinIndex>2</m_MinIndex>
        <m_MaxIndex>736</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Product</DimensionID>
        <PropertyID>Size Description1</PropertyID>
        <m_MinIndex>2</m_MinIndex>
        <m_MaxIndex>2205</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Time</DimensionID>
        <PropertyID>(All)</PropertyID>
        <m_MinIndex>1</m_MinIndex>
        <m_MaxIndex>1</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Time</DimensionID>
        <PropertyID>ID Season</PropertyID>
        <m_MinIndex>11</m_MinIndex>
        <m_MaxIndex>11</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Time</DimensionID>
        <PropertyID>ID Period</PropertyID>
        <m_MinIndex>39</m_MinIndex>
        <m_MaxIndex>39</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Time</DimensionID>
        <PropertyID>ID Week</PropertyID>
        <m_MinIndex>159</m_MinIndex>
        <m_MaxIndex>159</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Time</DimensionID>
        <PropertyID>ID Day</PropertyID>
        <m_MinIndex>1101</m_MinIndex>
        <m_MaxIndex>1107</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Time</DimensionID>
        <PropertyID>Dimdate</PropertyID>
        <m_MinIndex>1101</m_MinIndex>
        <m_MaxIndex>1107</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Time</DimensionID>
        <PropertyID>ID Year</PropertyID>
        <m_MinIndex>6</m_MinIndex>
        <m_MaxIndex>6</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Time</DimensionID>
        <PropertyID>Id Trdhalf</PropertyID>
        <m_MinIndex>8</m_MinIndex>
        <m_MaxIndex>8</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Time</DimensionID>
        <PropertyID>ID Qtr</PropertyID>
        <m_MinIndex>14</m_MinIndex>
        <m_MaxIndex>14</m_MaxIndex>
      </MapDataIndex>

Design Example

In this example, it is known that for a cube, users often want to query for a particular city and time range. It is common to partition by time, and to improve query time, we can also partition by geographical region. For the real customer cube, there is one city that represents more than 50% of the records. So we want to create a partition that looks graphically something like this:

2001, BigCity	2002, BigCity	2003, BigCity	2004, BigCity	2005, BigCity
2001, OtherCities	2002, OtherCities	2003, OtherCities	2004, OtherCities	2005, OtherCities

Remember that improving query time is by excluding partitions, and that is done by ensuring discontiguous ID ranges. The IDs are surrogate keys created during dimension processing. So for this example, it is required that during dimension processing, the query for Country, Province, and City, will result in BigCity being either before or after the other cities (and provinces and countries). For example, if it is in the middle, we will get these ranges:

BigCity partition: MinIndex = 500, MaxIndex = 500

OtherCites partition: MinIndex = 1, MaxIndex = 1000

Therefore a query with slice on SmallCity with ID = 123 would exclude the BigCity partitions, but a query with slice on BigCity would not be able to exclude OtherCities partitons, and would take longer.

The preferred result is a set of exclusive ranges (without overlap). Thus a query with slice on BigCity will exclude 50% of the partitions, and a query with slice on SmallCity will exclude 50% of the partitions. This is optimal.

BigCity partition: MinIndex = 1, MaxIndex = 1

OtherCites partition: MinIndex = 2, MaxIndex = 1000

So we know the goal, and should check the info.*.xml files after processing, to ensure they form exclusive ranges, and check representative MDX queries, and check the Query Subcube Verbose trace events.

Ordering the rows during dimension processing can be done using a named query as the source of the dimension in the DSV. One could define a different named query for each attribute. There is a chance this could hurt performance of dimension processing if any operation needed a join. Another approach is to substitute a view for the dimension table, to perform the appropriate sorting. Note that at present we have not actually done this, but we believe this approach will work.

Further Reading

Analysis Services 2005 Performance Guide

http://download.microsoft.com/download/8/5/e/85eea4fa-b3bb-4426-97d0-7f7151b2011c/SSAS2005PerfGuide.doc

AggManager, part of the SP2 samples.

http://www.microsoft.com/downloads/details.aspx?FamilyID=e719ecf7-9f46-4312-af89-6ad8702e4e6e&DisplayLang=en

[end]

Check out the exciting new SQL Server Best Practices website.....

thomasda — Sat, 09 Dec 2006 01:31:00 GMT

http://www.microsoft.com/technet/prodtechnol/sql/bestpractice/default.mspx

Get the real-world guidelines, expert tips, and rock-solid guidance to take your SQL Server implementation to the next level. Drawing on the extensive experience and expertise from respected developers and engineers at Microsoft as they walk you through the specifics on solving particularly difficult issues.