Microsoft SQL Server Development Customer Advisory Team : BI

Gemini - Self Service BI!

denny.lee — Sun, 16 Aug 2009 03:25:00 GMT

Gemini is the code name for the new breakthrough Self-Service Business Intelligence (BI) capabilities being delivered in the SQL Server 2008 R2 release. Gemini enables end users to build BI applications by integrating data from a variety of sources, modeling, refining and analyzing the data, adding business logic, building reports and visualizations and ultimately sharing it with their coworkers in an environment that is managed and secured by IT.

You should bookmark the Gemini Team Blog for the latest information on Gemini. As well, see SQL Server 2008 R2 | Self-Service Business Intelligence for more information. Once we start compiling best practices and lessons learned for Gemini, we will also publish more information on sqlcat.com.

Assigning surrogate keys to early arriving facts using Integration Services

tkejser — Wed, 13 May 2009 17:26:00 GMT

In data warehouses, it is quite common that fact records arrive with a source system key that has not yet been loaded in the dimension tables. This phenomena is known as “late arriving dimensions” or “early arriving facts” in Kimball terminology.

When you see a fact record that cannot be resolved to a dimension surrogate key, the typical solution is this:

Create a dummy member in the dimension table using the source system key
Assign a surrogate key to this dummy member
Use the newly create surrogate key and assign it to the fact record

If you use T-SQL to load the data warehouse, it means you have to pass over the input fact rows twice. First, you have to discover which keys are not present in the dimension (and create surrogates for them). Second, you will have to look at the input data again and use the newly generated surrogate keys to load the the fact table.

Using Integration Services, early arriving facts can be populated with just one pass over the source rows, which means less read I/O operations. Nice!

In project Project REAL, a script component is used to achieve this effect. If many of your dimension have early arriving facts, this creates a lot of copy/paste code. There is a cleaner solution that does not use script components.

There is a way handle early arriving facts without relying on script components. It is best illustrated with an example. Let us create these three tables:

/* The input table */
CREATE TABLE Stage_Fact
(
NK_A CHAR(10) NOT NULL /* The late arriving source system key */
)

/* The late arriving dimension table */
CREATE TABLE Dim_A
(
SK_A INT PRIMARY KEY IDENTITY(1,1) /* The surrogate key*/
, NK_A CHAR(10) NOT NULL /* The natural, source system key */
)

/* The final destination table */
CREATE TABLE Fact
(
SK_A INT NOT NULL /* Surrogate key from dimension */
)

Now, use this script to generate 16M rows in the input table and create a 9000 row dimension table:

/* Create some staging data */
INSERT Stage_Fact WITH (TABLOCK)
SELECT RIGHT(REPLICATE('0', 10) + CAST(K AS VARCHAR(10)), 10) AS NK_A
FROM (SELECT ABS(binary_checksum(*) % 10000) AS K
FROM sys.trace_event_bindings eb1
CROSS JOIN sys.trace_event_bindings eb2) AS stuff

/* Populate Dim_A with 90% of the keys from the fact table */
INSERT Dim_A WITH (TABLOCK) (NK_A)
SELECT DISTINCT NK_A FROM Stage_Fact
WHERE NK_A < '0000009000'

With this data, there will be 1000 late arriving dimension keys in Stage_Fact in around 1.8M rows. When we see a non-matched key in Stage_Fact, we want to generate a new surrogate key in Dim_A. But here is the catch: We only want to generate the surrogate once, and we do NOT want to do a roundtrip to the database the second time we see the same key.

Project Real uses a .NET hash table to track the generated keys and perform quick lookups the next time we see the key. But, we already have a fine hash table available without using script components: the lookup transformation. Let us see how we solve the early arriving fact problem with Integration Services, au natural:

The non-matched rows from Lookup SK_A go into the second lookup (New SK_A Cache). New SK_A Cache is where we want to handle the early arriving facts.

First, configure New SK_A Cache as a partial cache:

Now, we play a clever trick: Whenever a partial lookup cache first receives a non-matched row, it will call a SQL statement and fetch data to populate the lookup cache. The default is a SELECT statement, but it does not have to be a SELECT statement. We could replace it with a stored procedure that returns the same result as the SELECT. Actually, let us do exactly that:

Now, the FIRST time the partial lookup cache sees a early arriving fact, it will call Generate_SK_A. I have mapped the NK_A (the source system, natural key) column to the input parameter. To finish the trick, we just have to create a simple stored procedure that uses NK_A to lookup SK_A (the Surrogate Key), and if not found, create a new key:

CREATE PROCEDURE Generate_SK_A
@NK_A CHAR(10) /* The key to find a surrogate for */
AS
SET NOCOUNT ON

/* Prevent race conditions */
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE

/* Check if we already have the key (procedure is idempotent) */
DECLARE @SK_A INT
SELECT @SK_A = SK_A
FROM Dim_A
WHERE NK_A = @NK_A

/* The natural key was not found, generate a new one */
IF @SK_A IS NULL BEGIN
INSERT Dim_A (NK_A) VALUES (@NK_A)
SET @SK_A = SCOPE_IDENTITY()
END

/* Return the result.
IMPORTANT: must return same format is the SELECT statement we replaced */
SELECT @SK_A AS SK_A, @NK_A AS NK_A

Simple isn’t it?... No need to use any .NET script components here. Have a look at the attached files to study the technique further and you will be handling early arriving facts elegantly in no time.

Remember to move DTS2000 packages when upgrading msdb from 2005 to 2008

tkejser — Fri, 20 Mar 2009 14:06:00 GMT

Some customers prefer to upgrade to SQL Server 2008 by detaching all databases from their SQL Server 2005 instance and then reattaching them to a completely new SQL Server 2008 instance.

If you choose this upgrade strategy, you should be aware that you cannot detach an msdb database from a SQL Server 2005 instance and reattach it to a SQL Server 2008 instance. If you try, you will get an error message. Backup/restore will not work for msdb in this upgrade scenario either.

The msdb database may contain objects that are relevant to your installation – for example agent jobs and DTS 2000 packages. When upgrading, you should make sure you move all relevant objects in msdb. You can use tasks in Integration Services to move most of these objects. However, DTS 2000 packages cannot be moved using the Integration Services tasks. Instead, you must manually move the contents sysdtspackage.

The CSS team has a great blog entry describing how to move DTS 2000 packages to a SQL Server 2008 instance: How to copy DTS 2000 packages between servers (and from SQL 2000 to SQL 2005 and SQL 2008)

Using BIDS 2008 to validate Analysis Services 2005 cubes

tkejser — Thu, 26 Feb 2009 13:25:00 GMT

If you have experimented with cube design in SQL Server 2008, you will probably have run into the new AMO design warnings in Business Intelligence Development Studio 2008 (BIDS). We find that these warnings help customers a lot: they allow them to quickly analyze your cube for classic design mistakes and provide advice on how to correct them. In Analysis Services 2005 you need the best practice analyzer to get the same warnings.

Did you know that you can use BIDS 2008 to connect to Analysis Services 2005 cubes and immediately get the new 2008 AMO warnings feedback on your 2005 design? You can even save the cube back to Analysis Services 2005 with BIDS 2008. If you decide to save back the cube to a 2005 server you should first perform a backup of the 2005 cube.

Please note that you CANNOT use BIDS 2008 to save Integration Services or Reporting Services 2005 projects. If you open an Integration Services 2005 or Reporting Services 2005 project in BIDS 2008 it will be upgraded to the 2008 format instead.