Microsoft SQL Server Development Customer Advisory Team : Development & Programming

Using HierarchyID in SQL Server

kevincox — Thu, 27 Aug 2009 17:48:00 GMT

Implementing a hierarchy structure in a relational data base normally takes a bit of work. The new SQL Server data type for hierarchyID gives a good shortcut to the old methods, makes it faster to get a solution in place and makes it much easier to maintain.

MSDN has a good tutorial on HierarchyID showing the old method using a relational table design, followed by another design using the new HierarchyID. This article shows a good tip of how to easily get all descendents of a parent by adding a computed column using the GetAncestor(1) method that comes with the HierarchyID. It also has a complete list of other methods available.

Rather than repeat what is already on the MSDN tutorial, I will give examples of where it works well and where it doesn’t.

First, take a quick refresher course in the difference between a tree and a graph. These links are quick reads and will refresh your memory on those old courses you took at the university so many years ago. This is important to do before continuing because HierarchyID works well with trees but not graphs. Ok, that is a general statement for which there are some exceptions that I explain below.

WHERE IT WORKS WELL

The HierarchyID works well in projects where there is a hierarchical structure where each child has a single parent, like in a product/product classification hierarchy. In this example each product can only belong to one product class. You may have other levels, like product family where each product class can belong to only one product family.

Another, slightly more complex example would be a bill of materials structure for manufacturing a product. A part can belong to a subcomponent, which can belong to another subcomponent, which gets assembled into the final product. It is more complex than the first example because a part can belong to many subcomponents. HierarchyID may or may not work well in this case, it all depends on how many time a part can appear in the final product. If your graph is relatively unconnected, either due to relatively few parents or low links/node, then hierarchyId can be used for this type of graph using a primary key of (RootId int, Path hierarchyId).

All XML documents are trees and the HierarchyID works well with these projects.

WHERE IT DOESN’T WORK

1. When each child node in the graph has multiple parents. We tried this in a genealogy/ancestry project and discovered that to get this to work you would have to add a HierarchyID column for each parent. This is called a highly connected graph, where the number of paths is substantially more than the number of nodes. In this scenario, the traditional hierarchy design for relational databases works well, especially when combined with the CTE query pattern.

2. If subtree movement forms a substantial part of the workload. This is O(1) for parentiD/childId and O(subtree size) for hierarchyId. So if you are constantly updating the tree and cause the nodes to move, then the HierarchyID is not the best solution.

3. If subtree query isn’t a substantial part of the workload. This is O(1) for hierarchyId and O(subtree) for parentId/childId – but if not common this isn’t an advantage. In other words, if you don’t walk the tree often then it may not be worth using the Hierarchy data type.

Kevin Cox, Peter Carlin

Assigning surrogate keys to early arriving facts using Integration Services

tkejser — Wed, 13 May 2009 17:26:00 GMT

In data warehouses, it is quite common that fact records arrive with a source system key that has not yet been loaded in the dimension tables. This phenomena is known as “late arriving dimensions” or “early arriving facts” in Kimball terminology.

When you see a fact record that cannot be resolved to a dimension surrogate key, the typical solution is this:

Create a dummy member in the dimension table using the source system key
Assign a surrogate key to this dummy member
Use the newly create surrogate key and assign it to the fact record

If you use T-SQL to load the data warehouse, it means you have to pass over the input fact rows twice. First, you have to discover which keys are not present in the dimension (and create surrogates for them). Second, you will have to look at the input data again and use the newly generated surrogate keys to load the the fact table.

Using Integration Services, early arriving facts can be populated with just one pass over the source rows, which means less read I/O operations. Nice!

In project Project REAL, a script component is used to achieve this effect. If many of your dimension have early arriving facts, this creates a lot of copy/paste code. There is a cleaner solution that does not use script components.

There is a way handle early arriving facts without relying on script components. It is best illustrated with an example. Let us create these three tables:

/* The input table */
CREATE TABLE Stage_Fact
(
NK_A CHAR(10) NOT NULL /* The late arriving source system key */
)

/* The late arriving dimension table */
CREATE TABLE Dim_A
(
SK_A INT PRIMARY KEY IDENTITY(1,1) /* The surrogate key*/
, NK_A CHAR(10) NOT NULL /* The natural, source system key */
)

/* The final destination table */
CREATE TABLE Fact
(
SK_A INT NOT NULL /* Surrogate key from dimension */
)

Now, use this script to generate 16M rows in the input table and create a 9000 row dimension table:

/* Create some staging data */
INSERT Stage_Fact WITH (TABLOCK)
SELECT RIGHT(REPLICATE('0', 10) + CAST(K AS VARCHAR(10)), 10) AS NK_A
FROM (SELECT ABS(binary_checksum(*) % 10000) AS K
FROM sys.trace_event_bindings eb1
CROSS JOIN sys.trace_event_bindings eb2) AS stuff

/* Populate Dim_A with 90% of the keys from the fact table */
INSERT Dim_A WITH (TABLOCK) (NK_A)
SELECT DISTINCT NK_A FROM Stage_Fact
WHERE NK_A < '0000009000'

With this data, there will be 1000 late arriving dimension keys in Stage_Fact in around 1.8M rows. When we see a non-matched key in Stage_Fact, we want to generate a new surrogate key in Dim_A. But here is the catch: We only want to generate the surrogate once, and we do NOT want to do a roundtrip to the database the second time we see the same key.

Project Real uses a .NET hash table to track the generated keys and perform quick lookups the next time we see the key. But, we already have a fine hash table available without using script components: the lookup transformation. Let us see how we solve the early arriving fact problem with Integration Services, au natural:

The non-matched rows from Lookup SK_A go into the second lookup (New SK_A Cache). New SK_A Cache is where we want to handle the early arriving facts.

First, configure New SK_A Cache as a partial cache:

Now, we play a clever trick: Whenever a partial lookup cache first receives a non-matched row, it will call a SQL statement and fetch data to populate the lookup cache. The default is a SELECT statement, but it does not have to be a SELECT statement. We could replace it with a stored procedure that returns the same result as the SELECT. Actually, let us do exactly that:

Now, the FIRST time the partial lookup cache sees a early arriving fact, it will call Generate_SK_A. I have mapped the NK_A (the source system, natural key) column to the input parameter. To finish the trick, we just have to create a simple stored procedure that uses NK_A to lookup SK_A (the Surrogate Key), and if not found, create a new key:

CREATE PROCEDURE Generate_SK_A
@NK_A CHAR(10) /* The key to find a surrogate for */
AS
SET NOCOUNT ON

/* Prevent race conditions */
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE

/* Check if we already have the key (procedure is idempotent) */
DECLARE @SK_A INT
SELECT @SK_A = SK_A
FROM Dim_A
WHERE NK_A = @NK_A

/* The natural key was not found, generate a new one */
IF @SK_A IS NULL BEGIN
INSERT Dim_A (NK_A) VALUES (@NK_A)
SET @SK_A = SCOPE_IDENTITY()
END

/* Return the result.
IMPORTANT: must return same format is the SELECT statement we replaced */
SELECT @SK_A AS SK_A, @NK_A AS NK_A

Simple isn’t it?... No need to use any .NET script components here. Have a look at the attached files to study the technique further and you will be handling early arriving facts elegantly in no time.

FoxPro to SQL Server migration experience

kevincox — Fri, 25 Jan 2008 21:52:00 GMT

At first I was going to write a detailed white paper about the conversion effort. But someone has already done it. There is a very good guide at:

http://nationalcom.com/home/download/Conversion-VFP-SQLServer.pdf

Since FoxPro is nearing its end of life, the company I was working with needed to convert the system to newer technology. They chose Microsoft SQL Server and Microsoft .NET. There was over 1 terabyte of data being processed in FoxPro.

The project was fairly easy as far as the data conversion goes. There were a few decisions to make and one thing we learned about the difference in the NUMERIC data type between FoxPro and SQL Server.

The end result is pretty exciting. Some of the processing jobs that took over 1 day in FoxPro now take 25 minutes in SQL Server. The improvement is due to several facts: 1) The FoxPro code was doing row-at-a-time processing and the Transact-SQL code was written to do set processing; 2) FoxPro has a file limit of 2GB and with over 1TB of total data there were almost 1000 files to open and read through.

The most difficult part of the project was taking years worth of FoxPro code from different developers. The code had started with DOS FoxPro and had been converted to Windows FoxPro and upgraded by different people with different coding styles over the years. And as usual, the documentation was the code. J

Here are the notes from the data conversion effort. I’m only covering the data types that we had in our schema, which was not a very wide range. If you have more data types they are most likely covered in the guides noted above.

1. If you have MEMO data type in FoxPro you should use VARCHAR(MAX) in SQL Server. We could have used TEXT but we know that is being phased out in some future version in favor of VARCHAR(MAX). And VARCHAR(MAX) has better performance.

2. There are many DATE types in FoxPro. The decision was pretty easy to convert them all to DATETIME in SQL Server 2005. In SQL Server 2008 there are more DATE types so you will have to study which FoxPro types map to the new SQL Server 2008 types. DATETIME data was exported in character format using YYYYMMDDHHMMSS.

3. In FoxPro NUMERIC(5,2) allows a range of -99999 to 99999. In SQL Server using the same precision the range is -999.99 to 999.99. This was the only data type in our schema that needed to be altered at the data type level. So in SQL Server we had to use NUMERIC (7,2) to get the same range.

4. FoxPro LOGICAL fields were either exported as T/F or 0/1. They either went into a BIT data type in SQL Server or a TINYINT. The TINYINT was chosen if we wanted to do any calculations on the field in the future; i.e. SUM, AVG, etc.

Not knowing much about SQL Server, the database guys made a pretty good choice to dump the data to flat files in character mode, then used a simple Transact-SQL script to loop through the files and import them using the BULK INSERT command. A quote from one of the lead developers on the project: “We knew about Integration Services and could have used that but with our lack of experience with this tool we decided to use T-SQL scripts instead”. The scripts and data conversion took one person about 3 weeks to write and tune. This same person converted most of the other server side processing scripts to Transact-SQL in about 2 months.

Since FoxPro has a limit of 2GB file size and there was over 1TB of data in total, there were almost 1,000 files to convert. Everything was exported from FoxPro as character using a tab delimiter. It was determined that this was a safe delimiter after having to go through the memo fields to make sure there were no tab characters embedded.

Just as a side note, the FoxPro screens were all rewritten and reengineered using ASP.NET and C#. I will let someone else write about that experience if they wish.

Thanks to Spyros Christodoulou and Petros Hadjigeorgis from Nielsen EMEA for their assistance in writing this blog and for making the project successful.

kevin cox

New Best Practices Whitepaper published: "Best Practices for Migrating Non-Unicode Data Types to Unicode"

Sanjay Mishra — Tue, 22 Jan 2008 02:18:00 GMT

The database system in a company that is growing internationally must support multilingual characters in tables with Unicode data types. Existing databases that support only non-Unicode information must be migrated from non-Unicode data type to Unicode data type. This paper summarizes best practices for migrating database systems from non-Unicode to Unicode data type.

http://download.microsoft.com/download/d/9/4/d948f981-926e-40fa-a026-5bfcf076d9b9/SQL_bestpract_MigrationToUnicode.docx

New Best Practices Articles Published - Analysis Services Many-to-Many Dimensions: Query Performance Optimization Techniques

carl.rabeler — Sat, 22 Dec 2007 02:09:00 GMT

Many-to-many dimension relationships in SQL Server 2005 Analysis Services (SSAS) enable you to easily model complex source schemas and provide great analytical capabilities. This capability frequently comes with a substantial cost in query performance due to the runtime join required by Analysis Services to resolve many-to-many queries. This best practices white paper discusses three many-to-many query performance optimization techniques, including how to implement them, and the performance testing results for each technique. It demonstrates that optimizing many-to-many relationships by compressing the common relationships between the many-to-many dimension and the data measure group, and then defining aggregations on both the data measure group and the intermediate measure group yields the best query performance. The results show dramatic improvement in the performance of many-to-many queries as the reduction in size of the intermediate measure group increases. Test results indicate that the greater the amount of compression, the greater the performance benefits—and that these benefits persist as additional fact data is added to the main fact table (and into the data measure group).

http://www.microsoft.com/downloads/details.aspx?FamilyID=3494E712-C90B-4A4E-AD45-01009C15C665&displaylang=en

Precision Considerations for Analysis Services Users Whitepaper now available

denny.lee — Wed, 10 Oct 2007 08:28:00 GMT

For those of you whom want to know a little more about precision considerations for Analysis Services users, please check out this recently released whitepaper at http://www.microsoft.com/downloads/details.aspx?familyid=bae8beec-9892-4ecd-a9db-292254895f9c&displaylang=en.

This white paper covers accuracy and precision considerations in SQL Server 2005 Analysis Services. For example, it is possible to query Analysis Services with similar queries and obtain two different answers. While this appears to be a bug, it actually is due to the fact that Analysis Services caches query results and the imprecision that is associated with approximate data types. This white paper discusses how these issues manifest themselves, why they occur, and best practices to minimize their effect.

Authors: Denny Lee and Eric Jacobsen

New Best Practices Article

carl.rabeler — Wed, 12 Sep 2007 22:10:00 GMT

Scale-Out Querying with Analysis Services Using SAN Snapshots

SQL Server Best Practices Article: Identifying and Resolving MDX Query Performance Bottlenecks in SQL Server 2005 Analysis Services

carl.rabeler — Wed, 29 Aug 2007 10:06:00 GMT

Identifying and Resolving MDX Query Performance Bottlenecks in SQL Server 2005 Analysis Services

There are a variety of things that you can do to improve the performance of an individual MDX query. To begin, you must identify the source(s) of the performance bottlenecks in the execution of a poorly performing MDX query. To identify the source(s) of performance bottlenecks, you need to understand how to use the available Microsoft Windows and SQL Server 2005 Analysis Services troubleshooting tools to assist you with identifying bottlenecks and learn how to interpret the information revealed by these tools. This article provides the information about the available tools and demonstrates how to use the most common of these tools to identify and resolve MDX query performance bottlenecks with individual MDX queries.

Inconsistencies when using Drillthrough MDX with Perspectives

denny.lee — Tue, 19 Jun 2007 02:13:00 GMT

Author: Denny Lee

Contributors: John Desch, Kevin Cox

Reviewers: Robert Zare

DRILLTHROUGH MDX Overview

DRILLTHROUGH MDX is a very powerful way to drillthrough to your original data source of detail data that make up the aggregations that you are looking at. For more information on enabling drillthrough, please refer to T.K. Anand’s great article on Enabling Drillthrough in Analysis Services 2005 (http://technet.microsoft.com/en-us/library/ms345125(SQL.90).aspx). Using a real example, if you were to query the Adventure Works OLAP database for the [Internet Sales Amount] measure:

SELECT { [Measures].[Internet Sales Amount] } ON COLUMNS

FROM [Adventure Works]

The result would be that of: $29,358,677.22. You can also get the same result when you query the [Direct Sales] perspective (change the [Adventure Works] cube with the [Direct Sales] perspective in the above MDX statement).

With DRILLTHROUGH MDX statement, you can get the details that make up the above value of $29,358,677.22. Specifically, the MDX statement

DRILLTHROUGH

SELECT { [Measures].[Internet Sales Amount] } ON COLUMNS

FROM [Adventure Works]

will provide the dimension attributes and measure details that make up this aggregate value. Due to the large number of columns and rows included in the output, only the dimension members Promotion and Sales Territory Region and the Internet Sales Amount measure will be shown below:

[Promotion], [Sales Territory Region], …., [Internet Sales Amount], …

-------------------------------------------------------------------------

No Discount, Australia, …, 3399.99, …

No Discount, Australia, …, 3374.99, …

…, …, …, …, …

DRILLTHROUGH MDX ON PERSPECTIVES

But when you run the same drillthrough statement against the [Direct Sales] perspective,

DRILLTHROUGH

SELECT { [Measures].[Internet Sales Amount] } ON COLUMNS

FROM [Direct Sales]

you’ll notice that while all of the dimension attributes are included in the output, none of the measures are listed.

Yet, this output behavior is not consistent; if you were to run the query below against the [Sales Summary] perspective,

DRILLTHROUGH

SELECT { [Measures].[Sales Amount]} ON COLUMNS

FROM [Sales Summary]

The output is includes only four of the ten available measures in the perspective. To prove this latter point, execute the below statement against the [Sales Summary] perspective and you will see the ten measures listed.

select { [Measures].members} on columns from [Sales Summary]

As you can see from the MDX statements above, there is inconsistent output from the DRILLTHROUGH MDX statements in which some or none of the measures are included in the output. Yet, this issue does not come up when you query the cube directly. Does this mean you will have to run all of your drillthrough queries through the cube?

WORKAROUND TO SOLVE THIS

You certainly can do this and avoid the use of perspectives altogether will drilling down to the details. But if you want to use perspectives, the workaround to this problem is to list out all of the measures and dimension attributes you want to see within that perspective explicitly. An example in reference to the [Direct Sales] perspective is:

DRILLTHROUGH

SELECT { [Measures].[Internet Sales Amount] } ON COLUMNS

FROM [Direct Sales]

RETURN [Internet Sales].[Internet Sales Amount],[Internet Sales].[Internet Order Quantity],[Internet Sales].[Internet Extended Amount],[Internet Sales].[Internet Tax Amount],[Internet Sales].[Internet Freight Cost],[Internet Sales].[Internet Unit Price],[Internet Sales].[Internet Total Product Cost],[Internet Sales].[Internet Standard Product Cost],[$Internet Sales Order Details].[Carrier Tracking Number],[$Internet Sales Order Details].[Customer PO Number],[$Internet Sales Order Details].[Sales Order Number],[$Internet Sales Order Details].[Sales Order Line],[$Promotion].[Promotion],[$Delivery Date].[Date],[$Sales Territory].[Sales Territory Region],[$Product].[Product],[$Ship Date].[Date],[$Source Currency].[Source Currency Code],[$Date].[Date],[$Sales Reason].[Sales Reason],[$Customer].[Customer],[$Destination Currency].[Destination Currency Code]

By explicitly stating what is to be returned, the “missing” measures such as [Internet Sales Amount] will now show up with in your drillthrough to the [Direct Sales] perspective.

Please note, this is by design and will not change within Analysis Services 2008.

How to warm up the Analysis Services data cache using Create Cache statement?

Nicholas Dritsas — Thu, 26 Apr 2007 12:55:00 GMT

Goal

This document describes how to build Create Cache commands. Create Cache for Analysis Services (AS) was introduced in SP2 of SQL Server 2005. It can be used to make one or more queries run faster by populating the OLAP storage engine cache first.

Some customers have found certain queries benefit other later queries. For example, ascmd.exe could be used every hour to execute all queries in a directory keeping the cache ready for subsequent user queries. The other approach, which has been used, is to create a cache query for each user query. This is feasible if the MDX query is part of a report, then one simply adds another query that has the side effect of populating the cache, thereby speeding up the next query.

The root of the problem is that during a query, the AS Server does only local optimizations. Calculations, mixed granularities, and other more complex MDX statements can result in a chatty communication between the FE (Formula Engine) and the SE (Storage engine). In AS2000 and earlier, this was also a network round trip.

By issuing a cache statement, we can populate the cache with one or more subcubes that cover the regions of cube space that the query will actually need. We often find approximately the same time taken for each subcube query, so the effect can be dramatic overall.

With this methodology, the collection of MDX queries will appear as first executing inside the storage engine, and second inside the formula engine. In addition to reducing overall time, this can make it easier to predict the effect of multi-user load testing, because the first part uses 100% of all CPUs, and the second part uses 100% of one CPU.

Summary Steps

It is an iterative process. I would describe the identification of potential scenarios where create cache would help:

Run profiler
Run the query looking for Non-cache SE queries (query subcube filtered by subevent = 2)
Look at the total time of the query vis-à-vis of the sum of the times of the non-cache SE queries.
If:

They are pretty close and
There are many non-cache SE queries within the same range of time (not just 1 or 2 time consuming)

then create cache might help.

There could be variations, but here are one set of steps that have been used successfully:

1. Extract all MDX queries as separate files.

2. Add a Create Cache statement to correspond for every MDX query file.

3. Run Clear Cache, Create Cache, then the user query.

4. Verify that the Create Cache is effective and improve, if necessary.

5. Work on next query

6. When done with all queries, combine the Create Cache into 1 or more Create Cache queries.

7. Verify the combined Create Cache.

Detailed Description

Below are details about each of the steps.

1. Extract MDX queries as separate files

Placing the queries in separate MDX files, it can be faster to work one by one and verify if each query is handled correctly by Create Cache before moving on.

1. Start a trace.

2. Run the report.

3. Stop the trace.

4. Extract the queries. One way is with SQLProfiler, /File /Export /Extract SQLServer Analysis Serverices Events / Extract All Queries. This creates a text file, with each query on a separate line. When queries have multiple lines this might be confusing, but is easy to add an extra line or otherwise edit. Either copy each query to a separate file, or, for every query that is being worked on, comment out the other queries.

2. Add Create Cache for every MDX query

Below is an example of a Create Cache statement.

create cache

for [MyCube]

as (

{ [USA].[Oregon], [USA].[Colorado], [USA].[Florida], [USA].[Washington] }

* { [Measures].[mybasemeasure] }

* { [2006].children, parallelperiod( [Time].[Year], 1, [2006].[Q1].[Jan] ), YTD( parallelperiod( [Time].[Year], 1, [2006].[Q1].[Jan] ) ) }

* { [Products].[Shoes].children }

)

It should be apparent that it is basically a crossjoin of each dimension member that is specified in the query. Note that set expressions are allowed.

2.a. Add Specified Members

First add all members specified in the query. NOTE: Create Cache covers static analysis rather than dynamic. MDX with dynamic members will not benefit from this approach.

2.b. Add Calculated Members and Definitions

During execution, the Formula Engine can issue a subcube query which includes calculated members.

For example, if the cube has a calculated member:

[MyDim].[Calc123] as [MyDim].[A] + [MyDim].[B]

one should include the following members in Create Cache:

[MyDim].[A], [MyDim].[B] }

2.c. Account for Custom Rollups

During execution, there may be custom rollups that affect the calculation. For example, a custom rollup may involve QTD() or YTD(). This might be hard to detect since the actual calculations are stored in the relational database. Custom rollups can be viewed from BI Development Studio. The dimension must be processed (otherwise the calculations still only exist on the relational database).

1. Go to Solution Explorer, double-click on the dimension you want to examine.

2. Click on the Browser tab.

3. Find the icon for Member Properties, click on it. Select "Custom Rollup", "Custom Rollup Properties".

4. It will display the Custom Rollup (formula) and other properties (such as solve order) for each member.

For example, if calculations include YTD or PeriodsToDate, the member list should account for that. So, if the query includes:

[Date].[2006].[June],

ParallelPeriod( [Date].[2006].[June], [Date].[Year], -1 )

YTD( [Date].[2006].[June] )

the only calculations resulting in output are [Date].[2005].[June], [Date].[2006].[Jan] ..[Date].[2006].[June]. However, because of indirect relationships, you should also add the member set:

YTD( ParallelPeriod( [Date].[2006].[June], [Date].[Year], -1 )

3. Run Clear Cache, Create Cache, User Query

Now, it is time to run the Clear Cache statement and examine its effect. Note that the Cube ID can be specified or left blank to clear the entire database cache entries.

<DatabaseID>FoodCmp</DatabaseID>

<CubeID>SalesCube</CubeID>

</Object>

</ClearCache>

</Batch>

3.a. Connection string parameters

We recommend these parameters to be added at the connection string.

Provider=msolap.3;Datasource=MyServer;Initial Catalog=MyDatabase;Timeout=300;Disable Prefetch Facts=true;Cache Ratio=1

Provider: The name of the OLE DB provider, msolap.

Datasource: The name of the server. Might be named instance like MyServer\MyInstance.

Initial Catalog: The name of the database to use.

Timeout: Optional number of seconds for command timeout. This can help to avoid very long runs until the queries are optimized to run faster.

Disable Prefetch Facts: Optional new parameter to disable the Formula Engine heuristic that sends queries for possibly more data than it is requested.

Cache Ratio: Optional parameter to control the conversion from a specified set of members to the subcube query sent to the SE Storage Engine. Use 1 to send only the specified list.

3.b. Ways to Execute Queries

Because the connection string needs to be modified, SQL Server Management Studio cannot be used. There are several alternatives:

1. One could write a small program in C#.

2. ascmd.exe can execute queries and output time. The ascmd utility is part of the Yukon Samples kit – located here:

http://www.microsoft.com/downloads/details.aspx?FamilyID=e719ecf7-9f46-4312-af89-6ad8702e4e6e&DisplayLang=en

Then look in the C:\Program Files\Microsoft SQL Server\90\Samples\Analysis Services\Administrator\ascmd folder.

3. Use the mdx sample app from AS2000.

4. Verify Create Cache is Effective

The goal of using Create Cache is to isolate the storage engine and formula engine work and prevent the user query from executing subcube commands. The SE is multi threaded when the FE is single threaded. Therefore, one can just use Task Manager to verify if the SE is running.

During a good run, initially all the CPUs are at 100% since the SE is busy with the Create Cache command. Then, only one CPU reaches 100% because the FE is busy with the formulas and preparing the result set.

Task Manager is the general way to verify, but to be sure, create a trace and look for event Query Subcube, subevent 2 Non Cache. The cache subcube queries are fast and generally not an issue for AS. (In fact it shows how effective it is to create a cache first.)

4.a. Improve, if necessary

If the user query results in Task Manager CPU spikes (caused by subcube queries), one only needs to determine which members from the main query are not present in the Create Cache statement.

It might be helpful to look at trace information for Query Subcube Verbose. It might be that by comparing the ones for the Create Cache with the ones for the user query, some differences can be seen, and that could help discover which members should be added to the Create Cache query.

Another approach is to simplify the query until the problem does not happen. Try to use binary search to find the critical part of the user query that causes the extra subcubes.

4.b. Eliminate All Subcubes

This general technique works well when one can eliminate all subcubes. However, it might not be possible to know the set of members in advance (static analysis). If this is not true, for instance for complex calculations, then the Create Cache technique might not be as beneficial. Adding members that are not used can be cheap in some cases, for example to add one more product in an already large specified set. Or it might be expensive, for example to include another large set of data that otherwise would avoid entire partitions. One will need to experiment and measure the result, and because sometimes it will be a win and sometimes a loss, we recommend considering several user parameters.

5. Work on Next Query

6. After working with all the main queries, combine the Create Cache into 1 or More Create Cache statements

The best number of Create Cache queries depends on their structure (which dimension members involved, which measure group) and member sets.

1. Different measure groups should be separated unless the dimension members are consistent. Then, it is just a matter of convenience to combine, since under the covers the measure groups are physically separate.

2. If one query is covered or almost covered by another one, they should be combined.

3. If queries are disjoint or mostly disjoint, keep separate.

We have not experimented greatly, but here is one suggestion. Execute both scenarios, as separate and combined. Measure execution time and size of resulting DataCache (from perfmon counter). Use single combined Create Cache if the combination reduces time to 67% of separate commands and consumes space less than 3 times as much as separate.

7. Verify the combined Create Cache

Execute all queries, in this pattern:

1. Clear Cache

2. Create Cache (possibly many queries)

3. User queries

Verify the set of queries in the same way as each single query.

Contributors:

Eric Jacobsen

Cristian Petculescu

SSAS Partition Slicing

Eric Jacobsen — Tue, 06 Mar 2007 01:10:00 GMT

Summary:
SSAS uses partitions to contain cube data. Small cubes might use only one, but for non trivial cubes, cube designers will create partitions based on ease of managing data and to split groups of data. This document discusses how the server uses "slices" to examine partitions at query time. The slices are stored as a range of internal surrogate keys, and can be thought of as a very high level index.

Using more attributes in the partition definition can potentially improve performance by the ability to exclude partitions. To benefit user queries successfully requires some consideration of the internal way partition slices are used. Note that even though slices might not exclude partitons, we still expect the map indexes to be efficient at retrieving only neccessary data.

Slice description:
By "slice" we mean one or more members that limit the multidimensional space. Queries may have a "slice" as a result of including a set of members on an axis, or by placing members or a set in the where clause. Query slices are always a finite set of members. The formula engine may decompose MDX queries into many storage engine queries, which have a simple specification of granularity of each attribute, and a set of slices that limit the contents.

Cube partitions each have a slice for each attribute. This slice is either set by the administrator as part of the partition definition, or for MOLAP partitions is created during creation of the bitmap index for the partition. There is one slice per attribute in the partition. (Similarly there is one bitmap index per attribute in the partition.) Because the server automatically discovers the slice it is sometimes referred to as the "auto slice".

The partition slice value is stored in a file in the partition directory, with the name "Info.*.xml". The information stored is the minimum and maxiumum member IDs that are present in the partition, for each attribute. The member IDs are internal to the olap server, and are basically surrogate keys, created as a contiguous set of 32-bit integers starting with 1. This can have an impact when considering the min/max range way that partitions are created. It might be beneficial to change the SQL query for the dimension attributes so that the order in which attributes are returned matches the partitioning slicing scheme. Then the alignment of the internal surrogate keys helps the server at query time eliminate some partitions from being considered. See below the section Design Example.

The slice is created during index creation. Index creation is controlled by the number of rows, specified in the config file as <IndexBuildThreshold>, with a default of 4096. Partitions with fewer rows will not have indexes built -- and therefore will not have an "auto slice".

Optimal Number and Size of Partitions

There is not necessarily an optimal number or size of partitions, but some general guidelines exist. In Analysis Services 2005 Performance Guide (see link below), we suggest at most about 2000 partitions, and maximum of about 2 GB or 10-15 million rows in size. Some sites successfully use much larger partitions. For number of partitions, too few can miss opportunities to use slicing or can cause larger time windows for processing. Too many partitions might take more work to manage, cause some slowdowns of operations during processing (improved after SP2 was released), and can slow service startup time due to many more files on disk. Too small partitions might not offer any benefit. Too large partitions might increase the time window for processing past what is available.

Query Time Behavior

During query time all partitions are scanned for possible inclusion, by checking the query slices and partition slices for intersection. This is sequential, but is fast because it involves only integer range comparisons. Partitions that are included will be queried in parallel, and partitions that are excluded are not touched further.

There are SQLProfiler trace events for each partition that is used. Which trace event depends on whether the fact table or an aggregation, and each partition could have a separate aggregation definition.

Troubleshooting Information

It is important for very large cubes that a partition is not touched unless the query depends upon it. The information generally used to troubleshoot is:

Trace event Query Subcube Verbose. Check to see which attributes have slices.
Info.*.xml file, located in the partition. Make sure slices are defined for the attributes used in the query.
Trace events, can be used to check which partitions are being used. See below.

In SSAS2005 SP2 and earlier, the following behaviors may have a negative impact on the number of partitions touched.

Aggregations having fewer than <IndexBuildThreshold> rows are not excluded based on slice, even if a slice range exists for the fact table data. One possible work around is to lower the value of the <IndexBuildThreshold> configuration parameter in the server configuration file. However, this is not recommended as it may result in a large number of additional indexes, slowing processing performance.
Queries using "OR" slices do not use partition slices. The OR slice can be identified by a "+" in the trace event Query Subcube Verbose.

Both of these issues are expected to be addressed in a future Analysis Services release.

Trace Events

These are the relevant trace events to examine.

Query Subcube Verbose. This trace event is emitted after the internal storage engine subcube query has completed. The duration shows the total time for creating the datacache. Note that a query will only have one occurring at a time, and many partitions can be read from concurrently. So the start and end times will not overlap other Query Subcube Verbose events. Here is an example part of the output.

D:3 (Time) [1 8 52 * 0 0 0 0 0] => ((All)):[All] (Season):[Winter 2007] (Period):[JANUARY_2007] (Week):* (Day):0 (Dimdate):0 (Year):0 (Half):0 (Qtr):0

The numbers in square brackets [1 8 52 * 0 ...] have these meanings:

Single number - The internal surrogate key ID for the member participating in the slice. Note that the first element is often the All level, which by definition only has 1 member, and therefore its ID will always be 1. The member name is displayed.

'*' - The answer must have granularity for this attribute, and all members are used. There is no slice on this member.

0 - The answer should not include granularity for this attribute.

'+' - A slice is defined with more than 1 member. This is sometimes called an "OR" slice, because the equivalent SQL query would be of the form: where city = 'Seattle' or city = 'Redmond'. The list of members is not displayed. For simple MDX queries it is sometimes possible to infer the list.

Query Subcube. This trace event is not as readable for a human as Query Subcube Verbose. The format is a bitmap of attribute granularities, with dimensions delimited by a comma (",").

The same format of information is emitted by the Get Data From Aggregation trace event, and is also inserted into the QueryLog relational database table, if enabled in the configuration file. You can cut and paste this string directly into the AggManager sample to help create aggregations that match your queries, without or in addition to using the QueryLog relational table. (We wish to point out the format can be transferred -- not to suggest creating aggregations by just cutting and pasting. Designing the set of of aggregations is beyond the scope of this document.)

TextData from Query Subcube event : 00000000,000000000000000000000000,01110000

In this example, there are 3 dimensions, and the query only has granularity on attributes in the 3rd dimension. (Which for this example happened to be the Time dimension.)

Get Data From Aggregation. This trace event is emitted when reading from a partition, in an aggregation (not from the lowest level fact data). It is not emitted if the partition can be excluded based on the partition's "auto slice".

Note that the start and end time will typically overlap the start end end time of other events, because storage engine reads partition data in parallel, and the time will be contained within the time range for Query Subcube and Query Subcube Verbose.

The ObjectPath trace column tells what partition is being read. Note that Query Subcube and Query Subcube Verbose also have ObjectPath, but are missing the partition information since the query represents the union of all its partitions. The format is as follows. Note that for named instances the server name might be of the form MachineName\ServerInstanceName.

ServerName . DatabaseID . CubeID . MeasureGroupID . PartitionName

The TextData column contains the name of the aggregation used and the list of attribute granularity for the aggregation used. It will always "cover" (be larger than) the granularity of the Query Subcube. Here is an example:

TextData from Query Subcube event : 00000000,000000000000000000000000,01110000

TextData from Get Data From Aggregation event: Aggregation 1 00000000,000000000000000000000000,01110111

Progress Report Begin, Progress Report End These trace events are emitted when reading from a partition, in the fact data (not from an aggregation). It is not emitted if the partition can be excluded based on the partition's "auto slice".

The ObjectName trace column tells what partition is being read. The TextData column also tells the same information. Here is an example:

TextData from Progress Report Begin event : Started reading data from the 'MyPartitionName' partition.

Example of Partition Slice Information

The exact MDX query is unimportant, so we will just look at the Query Subcube Verbose and some partition information.

Query Subcube Verbose, from SQLProfiler trace:

Dimension 0 [Location] (0 0 0 0 0 0 0 0) [Dimlocation]:0 [Chain]:0 [Division]:0 [Region]:0 [Area]:0 [Store]:0 [Location Type]:0 [Grouped Stores Ind]:0
Dimension 1 [Product] (0 2 2 14 137 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0) [Dimproduct]:0 [Company]:[C SAMPLE] [Brand]:[B 1 SAMPLE] [Group]:[G 105 SAMPLE] [Dept]:[D 1125 SAMPLE] [Category]:0 [Range]:0 [Style]:0 [Colour]:0 [Size 1]:0 [Size 2]:0 [Size]:0 [Season Ind]:0 [Replenishment]:0 [Clearance]:0 [RSP]:0 [Primary Supplier]:0 [Supplier Color]:0 [SKU]:0 [Size Indicator1]:0 [VPN]:0 [Sku Description]:0 [Latest Cost]:0 [Original RSP]:0
Dimension 2 [Time] (0 * * * + 0 0 0) [Dimdate]:0 [Week]:* [Period]:* [Season]:* [Day]:+ [Year]:0 [Qtr]:0 [Half]:0

The "0" means no granularity for that attribute.

The "*" means include granularity for that attribute, with all members.

The "+" means include granularity for that attribute, with a set of members.

A number means include granularity for that attribute, with one specific member. The number is the internal surrogate key.

This subcube query included a slice on a set of days. Though the set members are not included in the trace event, for the actual query the set of [Time].[Day] for this example was { 1465, 1466, ... 1492 }.

So we can expect for this query, that partitions slicing by [Product] or [Time] would be effective at reducing query time by using slice information as a high-level index and reducing the work for the query.

SQLProfiler Trace information:

For this example, aggregations were able to answer the subcube query. So the relevant trace events are Progress Report Begin, and Progress Report End. Here are the relevant columns for one trace event. (Note that in SQLProfiler, this is displayed horizontally as one row.)

EventClass: Progress Report Begin

EventSubclass: 14 - Query

TextData: Started reading data from the 'Aggregation 9' aggregation.

DatabaseName: Sample

ObjectName: Aggregation 9

ObjectID: Aggregation 9

ObjectPath: MyMachine.Sample.Sample SMDB.Fact Sales SKU Store Day 0.Sales_W200639.Aggregation 9

Note that the partition name is part of the ObjectPath, a trace column that is disabled by default.

Here is a subset of info.*.xml for a customer partition year 2005 week 36. We can see that if the query slice contains any of the Product or Time properties, partitions should be excluded. Note that some ranges are narrow (few members), and some ranges are broad. For example, the range for [Product].[Colour] is 2...736, which is not surprising since the product sold in this week probably contains every color. If there was a goal of partitioning also by color or some other product attribute, that would help queries that sliced by that attribute.

For the example query subcube, we expect it to be excluded based on [Time].[Day]. For this example, it actually was excluded due to the set of days not in the inclusive range 1101...1107 for the property Day.

      <MapDataIndex>
        <DimensionID>Product</DimensionID>
        <PropertyID>ID Company</PropertyID>
        <m_MinIndex>2</m_MinIndex>
        <m_MaxIndex>2</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Product</DimensionID>
        <PropertyID>ID Brand</PropertyID>
        <m_MinIndex>2</m_MinIndex>
        <m_MaxIndex>2</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Product</DimensionID>
        <PropertyID>ID Group</PropertyID>
        <m_MinIndex>2</m_MinIndex>
        <m_MaxIndex>21</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Product</DimensionID>
        <PropertyID>ID Style</PropertyID>
        <m_MinIndex>22</m_MinIndex>
        <m_MaxIndex>325486</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Product</DimensionID>
        <PropertyID>ID SKU</PropertyID>
        <m_MinIndex>122</m_MinIndex>
        <m_MaxIndex>1307861</m_MaxIndex>
      </MapDataIndex>
        <DimensionID>Product</DimensionID>
        <PropertyID>Colour</PropertyID>
        <m_MinIndex>2</m_MinIndex>
        <m_MaxIndex>736</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Product</DimensionID>
        <PropertyID>Size Description1</PropertyID>
        <m_MinIndex>2</m_MinIndex>
        <m_MaxIndex>2205</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Time</DimensionID>
        <PropertyID>(All)</PropertyID>
        <m_MinIndex>1</m_MinIndex>
        <m_MaxIndex>1</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Time</DimensionID>
        <PropertyID>ID Season</PropertyID>
        <m_MinIndex>11</m_MinIndex>
        <m_MaxIndex>11</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Time</DimensionID>
        <PropertyID>ID Period</PropertyID>
        <m_MinIndex>39</m_MinIndex>
        <m_MaxIndex>39</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Time</DimensionID>
        <PropertyID>ID Week</PropertyID>
        <m_MinIndex>159</m_MinIndex>
        <m_MaxIndex>159</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Time</DimensionID>
        <PropertyID>ID Day</PropertyID>
        <m_MinIndex>1101</m_MinIndex>
        <m_MaxIndex>1107</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Time</DimensionID>
        <PropertyID>Dimdate</PropertyID>
        <m_MinIndex>1101</m_MinIndex>
        <m_MaxIndex>1107</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Time</DimensionID>
        <PropertyID>ID Year</PropertyID>
        <m_MinIndex>6</m_MinIndex>
        <m_MaxIndex>6</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Time</DimensionID>
        <PropertyID>Id Trdhalf</PropertyID>
        <m_MinIndex>8</m_MinIndex>
        <m_MaxIndex>8</m_MaxIndex>
      </MapDataIndex>
      <MapDataIndex>
        <DimensionID>Time</DimensionID>
        <PropertyID>ID Qtr</PropertyID>
        <m_MinIndex>14</m_MinIndex>
        <m_MaxIndex>14</m_MaxIndex>
      </MapDataIndex>

Design Example

In this example, it is known that for a cube, users often want to query for a particular city and time range. It is common to partition by time, and to improve query time, we can also partition by geographical region. For the real customer cube, there is one city that represents more than 50% of the records. So we want to create a partition that looks graphically something like this:

2001, BigCity	2002, BigCity	2003, BigCity	2004, BigCity	2005, BigCity
2001, OtherCities	2002, OtherCities	2003, OtherCities	2004, OtherCities	2005, OtherCities

Remember that improving query time is by excluding partitions, and that is done by ensuring discontiguous ID ranges. The IDs are surrogate keys created during dimension processing. So for this example, it is required that during dimension processing, the query for Country, Province, and City, will result in BigCity being either before or after the other cities (and provinces and countries). For example, if it is in the middle, we will get these ranges:

BigCity partition: MinIndex = 500, MaxIndex = 500

OtherCites partition: MinIndex = 1, MaxIndex = 1000

Therefore a query with slice on SmallCity with ID = 123 would exclude the BigCity partitions, but a query with slice on BigCity would not be able to exclude OtherCities partitons, and would take longer.

The preferred result is a set of exclusive ranges (without overlap). Thus a query with slice on BigCity will exclude 50% of the partitions, and a query with slice on SmallCity will exclude 50% of the partitions. This is optimal.

BigCity partition: MinIndex = 1, MaxIndex = 1

OtherCites partition: MinIndex = 2, MaxIndex = 1000

So we know the goal, and should check the info.*.xml files after processing, to ensure they form exclusive ranges, and check representative MDX queries, and check the Query Subcube Verbose trace events.

Ordering the rows during dimension processing can be done using a named query as the source of the dimension in the DSV. One could define a different named query for each attribute. There is a chance this could hurt performance of dimension processing if any operation needed a join. Another approach is to substitute a view for the dimension table, to perform the appropriate sorting. Note that at present we have not actually done this, but we believe this approach will work.

Further Reading

Analysis Services 2005 Performance Guide

http://download.microsoft.com/download/8/5/e/85eea4fa-b3bb-4426-97d0-7f7151b2011c/SSAS2005PerfGuide.doc

AggManager, part of the SP2 samples.

http://www.microsoft.com/downloads/details.aspx?FamilyID=e719ecf7-9f46-4312-af89-6ad8702e4e6e&DisplayLang=en

[end]

Check out the exciting new SQL Server Best Practices website.....

thomasda — Sat, 09 Dec 2006 01:31:00 GMT

http://www.microsoft.com/technet/prodtechnol/sql/bestpractice/default.mspx

Get the real-world guidelines, expert tips, and rock-solid guidance to take your SQL Server implementation to the next level. Drawing on the extensive experience and expertise from respected developers and engineers at Microsoft as they walk you through the specifics on solving particularly difficult issues.

Best SQL Server 2005 MDX Tips and Tricks - Part 1

Nicholas Dritsas — Thu, 12 Oct 2006 14:42:00 GMT

Overview

SQL Server’s 2005 Analysis Services has introduced several changes to the MDX queries syntax that can lead to better performance than the equivalent AS 2000 queries.

Here is a list of tips and tricks as well as best practices on how to get better performance from your MDX queries in AS2005.

Details

For filtering a set, use Filter inside Crossjoin vs the other way around

Filter a set and then use it in the Crossjoin. Filter function materializes the set and iterates through the set to check condition to build new set.

Avoid:

filter(NECJ({set1},{set2}),..)

Use:

NECJ(filter({set1},...),{set2})"

Use Rank() over Intersect() function to check if a member exists in a set

The disadvantage of using Intersect() to determine if a member exists in a set is because it treats the member as a set and can not use a better plan in the evaluator.

Avoid:

iif(intersect({ACTUALS_DAYS_SET},{[TIME DIM].[Time Main].currentmember}).count)>0

Use:

iif(rank([TIME DIM].[Time Main].currentmember, {ACTUALS_DAYS_SET})>0

Use Curly Braces in crossjoin with single member

When doing Crossjoins always use sets (add curly braces around single member if used in crossjoin).

Avoid:

Sum( [FINANCIAL VERSION DIM].[Financial Version].[Financial Version Type].&[WSLT]

*{[GROUP STATUS DIM].[Group Status].[Group Status Name].[cancel]

,[GROUP STATUS DIM].[Group Status].[Group Status Name].[turn down]},measure)

Use:

Sum(

{[FINANCIAL VERSION DIM].[Financial Version].[Financial Version Type].&[WSLT]}

*{[GROUP STATUS DIM].[Group Status].[Group Status Name].[cancel]

,[GROUP STATUS DIM].[Group Status].[Group Status Name].[turn down]},Measure)

Avoid unnecessary .CurrentMember in calculations.

It is not a good practice to use .CurrentMember when it is not required in the calculations. The formula engine can generate a better query plan if MDX does not use “.CurrentMember” to select the current member of a dimension. CurrentMember is implied and does not need to be explicitly included in the syntax.

No Need to use [TIME DIM].[Time Main].[Year].currentmember in the following MDX.

WITH

MEMBER [Measures].[M] as

'([TIME DIM].[Time Main].[Year].currentmember

,[FINANCIAL VERSION DIM].[Financial Version].[Financial Category].&[ACTL])'

select

{[TIME DIM].[Time Main].[Year].&[2005].members}

descendants([GROUP EVENT DIM].[Group Event].[Hotel].&[12]

,[GROUP EVENT DIM].[Group Event].[Group Event])}

Use Exists Function

Exists function should be used where ever possible instead of filter on member properties.

Avoid Lookup

Avoid using lookup function. Try to find out if the cube structure can be modified to have the measure available in the same cube.

Use Minus over Filter for a single member

When filtering out a single member from the set use minus over filter function

Avoid:

filter({set},.Currentmember <> "UNKN")

Use :

( {set} minus {&[UNKN] member})

Do not use calculated members that are constants

Although there may not be any difference for a simple example, when combined with other calculations, it can cause a more complicated execution plan inside the server. Note that when using parallelperiod function, often the function evaluates to a constant, so if it is known in advance that is preferred because the engine does not check for certain patterns that are known to be constant. It can be faster for a UI tool to send a first query to resolve parallelperiod (without other calculations), then substitute into the original query, rather than sending one more complicated query.

Avoid:

with

member [a].[NiceName] as '[a].[123]'

member [Time].[YearBefore] as 'parallelperiod( [Time].[year], 1, [Time].[2006].[jan] )'

select { [a].[NiceName] } on 0,

{ [Time].[2006].[jan], [Time].[YearBefore] } on 1

from [MyCube]

Use:

select { [a].[123] } on 0,
{ [Time].[2006].[jan], [Time].[2005].[jan] } on 1
from [MyCube]

How to check if the cell is empty?

Usually empty cell is checked for avoiding division by zero or for checking if value is missing (NON EMPTY analysis).

Check for non empty to avoid division by zero

For a/b calculations

IIF(b=0,NULL,a/b)

Empty cells are treated as zero in arithmetic operations.

For checking empty cells (Non Empty Analysis)

Filter([dimension].[hierarchy].member.members, isEmpty(dim.member))

This invokes MDX function IsEmpty, which, as the name suggests, checks whether cell value (in this case at coordinate b) is empty or not. Note, that if the cell value is empty, it is treated as number zero in arithmetic operations, however, it is possible, that b had value zero, which is not empty! Therefore check for IsEmpty is appropriate when the user wants to differentiate empty or missing values from the existing values (for example in NON EMPTY like analysis), but is not appropriate for checks in division by zero.

Note: Dot not ever use IS operator (ie: IIF(b IS NULL, NULL, a/b)) to check if the cell value is empty. The IS operator checks if the member b exists or not.

Miscellaneous design tips

Avoid assigning values like 0, Null, “N/A”, “-“ to cells that would remain empty otherwise. Use Format_String instead, for custom UI formatting of values.
Avoid redundant Sum/Aggregate calculations in situations where default/normal cell value aggregation would do.
Try to avoid IIF. If have to use IIF, see if it’s possible to write it in such a way that one of the branches (then/else) is written “null”.
Prefer using static literal hierarchy and member references (e.g. Measures.Sales instead of Dimensions(0).Sales, Product.[All] instead of Product.Members.Item(0) etc.).
Avoid using LinkMember, StrToSet, StrToMember, StrToValue.
Prefer using the new calculation Non_Empty_Behavior optimization hint instead of writing calculation expressions of the form Aggregate(NonEmptyCrossjoin(Descendants(…, Leaves) …).
If wanting to get the value of the current cell, consider using an explicit measure name instead of Measures.CurrentMember.
When writing calculation expressions like “expr1 * expr2”, make sure the expression sweeping the largest area/volume in the cube space is on the left side. For instance, write “Sales * ExchangeRate” instead of “ExchangeRate * Sales”, and “Sales * 1.15” instead of “1.15 * Sales”.
Consider replacing simple “Measure1 + Measure2” calculations with computed columns (additional measures) in the DSV or in the SQL data source.
Instead of writing expressions like Sum(Customer.City.Members, Customer.Population.MemberValue), consider defining a separate measure group on the City table, with a Sum measure on the Population column.

Author: Nicholas Dritsas

SQL Server Sequence Number

howardy — Tue, 11 Apr 2006 01:32:00 GMT

Simulating Sequence Objects in SQL Server

Many applications need sequentially incremental number as unique/primary key of records. SQL Server 2005 today supports identity column as the primary mean to general sequence number, which generates the sequence number upon the execution of DML (insert) or bulk insert. The value of the insertion is known (when using insertion DML), post the execution of the DML. In some implementations, the sequence numbers (or next unique incremental number) need to be generated prior to the DML operation. For example, if sequence number can be retrieved prior to the DML operation, it can be easily implemented in situations where multiple DML (inserts, or even updates) using the same sequence number can be performed, and data can be insert into different tables, for example, parent-child, or based on the sequence value, data can be insert into different tables.

SQL Server 2005 Migration Assistant (SSMA) provides a different mechanism for sequence number generation. SSMA’s approach is to use a single table to hold all the sequence numbers. Each sequence number object is represented by a single row of record, which holds the property of the sequence, e.g. sequence name, current value, increments…etc. The generation of the next value is invoked by an extended stored procedure in order to minimize the impact on application concurrency – the execution of extended procedure is not within the transaction scope of the DML statement, and it holds very short duration locks on the row that represents the sequence number.

This BLOG describes two more methods of sequence number generation.

Option 1

This option is easy to use and maintain, but it is suitable only with lower volume invocations. It is also one of the widely used approaches, and often misused. This method uses a single table to hold all the sequences needed for the application, and each sequence is represented by one single record, thus serializes operation requesting next value of the same sequence number.

1. Create the following objects

Table AllSequences holds the information of all the sequences. As a simple example, the table and subsequent stored procedure is kept quite simple. Further enhancement is certainly possible based on the needs – for example, additional information of sequence such as maximum range, minimum range, loopback(yes/no) can be added.

· SeqName – name of the sequence

· Seed – seed value of the sequence

· Incr – incremental value of the sequence

· Currval – current value of the sequence

Create table AllSequences (

SeqName nvarchar(255) primary key, -- name of the sequence

Seed int not null default(1), -- seed value

Incr int not null default(1), -- incremental

Currval int

)

The following stored procedure is used to create a sequence. The following input parameter is expected:

· SeqName – name of the sequence

· Seed – seed value of the sequence

· Incr – incremental value of the sequence

create procedure usp_CreateNewSeq

@SeqName nvarchar(255),

@seed int = 0,

@incr int = 1

begin

declare @currval int

if exists (

select 1 from AllSequences

where SeqName = @SeqName )

begin

print 'Sequence already exists.'

return 1

end

if @seed is null set @seed = 1

if @incr is null set @incr = 1

set @currval = @seed

insert into AllSequences (SeqName, Seed, Incr, CurrVal)

values (@SeqName, @Seed, @Incr, @CurrVal)

end

The following stored procedure is used to generate the next value of the specified sequence through input parameter SeqName.

create procedure usp_GetNewSeqVal

@SeqName nvarchar(255)

begin

declare @NewSeqVal int

set NOCOUNT ON

update AllSequences

set @NewSeqVal = CurrVal = CurrVal+Incr

where SeqName = @SeqName

if @@rowcount = 0 begin

print 'Sequence does not exist'

return

end

return @NewSeqVal

end

2. To create the new sequence, for example:

usp_CreateNewSeq N'TestSeq'

3. To retrieving the new sequence value, for example for sequence “TestSeq”

usp_GetNewSeqVal N'TestSeq'

One thing to be careful is to keep the execution of the stored procedure usp_CreateNewSeq outside the scope of subsequent transaction which uses the sequence number generated. This can minimize impact of blocking.

Typical usage

Declare @NewSeqVal int

Execute @NewSeqVal=usp_GetNewSeqVal @seqname=N'TestSeq'

print @NewSeqVal

Option 2

This method is intended for higher-throughput environments – such as thousands of transactions per second. A table with identity column is created for every sequence. Since the generation of identity value is lightweight, and not within the transaction scope of subsequent DML, this method can significantly improve the concurrency over the first method. In addition, it also consumes less resource than extended procedure.

1. Create one table with the following structure for each sequence, assuming seed 1, with increment of 1).

create table <tablename> (

SeqID int identity(1,1) primary key,

SeqVal varchar(1)

)

2. Create the following stored procedure. A delete operation is performed to remove the rows in the sequence table to reduce maintenance. This obviously adds some load to the transaction however it should be quite minimal. As an alternative, for higher throughput, the delete statement can be removed and a maintenance job can be scheduled during off-hour to purge the records in the table (truncate should not be used since it resets the identity value).

create procedure GetNewSeqVal_<tablename>

begin

declare @NewSeqValue int

set NOCOUNT ON

insert into <tablename> (SeqVal) values ('a')

set @NewSeqValue = scope_identity()

delete from <tablename> WITH (READPAST)

return @NewSeqValue

end

3. To retrieve the new value in a result set, simply execute the following:

GetNewSeqVal_<tablename>

Typical usage

Declare @NewSeqVal int

Exec @NewSeqVal = GetNewSeqVal_<tablename>

Among the two options described above, Option 1 is widely used in different variations but prone to blocking/deadlocking. Option 2 improves concurrency significantly with a simple approach.

Howard Yin

Managing Schema Changes (Part 2)

thomasda — Fri, 31 Mar 2006 19:10:00 GMT

SQLCAT Blog: Managing Schema Changes (Part 2)

Question: What are best practices for managing schema changes in SQL Server 2005? What are the fastest, least intrusive techniques for high availability?

Answer: In Part 1, we outlined the behavior of SQL Server 2005 schema changes. In Part 2, we will look at best practices techniques to de-couple the batch updates required (see “Managing Schema Changes Part 1”) for data type and most length changes. These techniques allow us to make schema changes quickly and with least impact to performance and concurrency for large tables like the Billion row scenario.

Scenario: Let’s assume we have a 5 character ZipCode column that we want to increase to 10 character format e.g. 12345-0001. We can change the ZipCode length from char(5) to char (10) using:

alter table alter column ZipCode Char(10)

However, this will incur a concurrency crippling batch update (as seen in Profiler) to change the column length in our Billion row table scenario. If you’d like to capture this in Profiler, see “Managing Schema Changes (Part 1).

Best Practice

The use of a NULL property (excluding DEFAULT WITH VALUES of course) is a best practice insofar as columns can be added quickly. In the case where you do need to set a value to the column, the NULL property also gives us the flexibility of de-coupling the update from the alter table statement.

An approach to accomplish this:

Add a char(10) column, NewZip
Set piecemeal batch size
Populate NewZip using an explicit conversion of ZipCode
Drop old column ZipCode.
Rename NewZip to ZipCode (optional)

Step 1: Add new column

alter table MyTable

add NewZip char(10) NULL

Upon the fast completion of this alter table (about 1 second in my tests), NewZip contains NULLs for all rows in the existing table. You may need to populate NewZip with the values from another column or other values. In step 2, SET ROWCOUNT is used to minimize the concurrency impact of write locks by limiting the number of rows that are affected at one time. An alternative to SET ROWCOUNT is a WHERE clause that specifies certain key ranges, limiting the number of rows affected at one time.

Step 2: Set Batch Size for update

--- (2) set piecemeal batch size

set nocount on

SET ROWCOUNT 1000

Next, the piecemeal update is repeatedly done until there are no NULL values in NewZip. This example sets all rows using the char(5) ZipCode column. Alternately, you can selectively set the value of an added column using a lookup or join.

Step 3: De-coupled Piecemeal Update

declare @rowcount int, @batchcount int, @rowsupdated int

select @rowcount = 1,@batchcount = 0, @rowsupdated = 0

while @rowcount > 0

begin

begin tran

update MyTable

set NewZip = ZipCode

where NewZip is NULL

set @rowcount = @@rowcount

select @rowsupdated = @rowsupdated + @rowcount

if @rowcount > 0 select @batchcount = @batchcount + 1

commit

end

Step 4: drop the char(5) column ZipCode (optional)

alter table MyTable

drop column ZipCode

Step 5: Rename the NewZip column to ZipCode (optional)

exec sp_rename 'MyTable.NewZip','ZipCode','COLUMN'

--- Observe the data type for ZipCode is now char(10)

sp_help MyTable

Trigger for ZipCode

What happens if a user changes a customer ZipCode AFTER the NewZip update? If the user application updates ZipCode char(5) during the piecemeal update process, NewZip may have already been updated with the old zip code value, and thus is out of synch with ZipCode. An update trigger can be used to keep ZipCode and NewZip in synch during the piecemeal update process. If you intend to drop the old char(5) ZipCode (in Step 4), and then rename NewZip to ZipCode (in Step 5), the update trigger will no longer be needed after the piecemeal update.

Explicit data type conversion

Some data type changes require explicit conversion. That is, they cannot be implicitly converted using alter table alter column. For example, an explicit conversion is required to change a datetime datatype to an int or numeric(8) YYYYMMDD format. This scenario can be handled using a convert (or datepart) function in exactly the same manner as the piecemeal update scenario above.

Dropping columns

Dropping columns is a schema only operation. There are no concurrency issues unless you move the object to reclaim space from the dropped column.

Unique and Primary Key Constraints

Adding Unique and Primary Key constraints involves reads, sorts, and writes. By default UNIQUE constraints add nonclustered indexes while PRIMARY KEY adds a clustered index. Obvious performance and concurrency issues would occur when you add a PRIMARY KEY (and default CLUSTERED INDEX) to a large table that already has UNIQUE (and nonclustered index) constraints. This is because adding a Primary Key (and clustered index) would rebuild the nonclustered indexes replacing RIDs (row IDs) with Primary Keys.

Alter table, create and alter index includes ONLINE options for managing unique and primary key constraints and indexes. Where possible, for best performance, highest concurrency and availability, ONLINE options should be used for managing indexes. Otherwise, it would wise to postpone these operations to batch maintenance windows.

Conclusions and best practices:

SQL Server 2005 does support schema changes however some schema changes are faster than others. The fastest schema changes occur when:

Adding column with NULL property
Adding column with NULL property and DEFAULT
Changing NOT NULL to NULL property
Adding DEFAULT constraint
Dropping CHECK or DEFAULT constraint

A schema change best practice for performance and concurrency of large tables includes adding columns with NULL properties. In the case where you need to set a value to the newly added column, starting with a NULL property provides the flexibility where the update can be de-coupled from the alter table statement. Far preferable to a batch update that can significantly affect concurrency on large tables; a piecemeal update can be used to limit the number of rows updated at one time. One technique to limit the number of rows is SET ROWCOUNT. This greatly reduces (or effectively eliminates) the blocking impact of the piecemeal update. If it is important to disallow NULL values after the piecemeal update completes, the application would be have to enforce this.

The inclusion of CHECK constraints requires a read of the entire table to enforce the CHECK constraint. The accompanying table scan can impact write activity for a large table.

Other schema changes present performance and concurrency challenges for large tables due to accompanying batch update operations. Those include:

Adding NULL column with DEFAULT and WITH VALUES clauses
Adding NOT NULL column
Changing NULL to NOT NULL property

Using the Piecemeal Update best practice with NULL columns allows us to avoid the undesirable associated activities occurring with some schema changes such as unintended reads and writes, enabling us to utilize the fastest, least intrusive techniques for high concurrency and availability.

Tom Davidson

SQL Server Customer Advisory Team