Russell Christopher's Semi-Useful BI Musings : SQL Server Integration Services

SQL Server 2008 "Katmai" CTP5 released

russch — Mon, 19 Nov 2007 22:18:00 GMT

Just got an invite from Microsoft Connect to download the November CTP of SQL 2008....you can, too!

http://www.microsoft.com/downloads/details.aspx?FamilyId=3BF4C5CA-B905-4EBC-8901-1D4C1D1DA884&displaylang=en

Alerting without SQL Server Notification Services

russch — Thu, 25 Oct 2007 02:41:00 GMT

Last week I worked with a customer who wanted to give users the ability to create and subscribe to alerts for changing KPI Values. This is a pretty standard scenario, and one we covered with BSM 2005. However, with the “loss” of Notification Services in SQL 2008, PerformancePoint server does not include alerting functionality out of the box (which stinks, but that’s another rant).

So, the customer was interested in creating a simple “SQLNS-less” alerting system that could send users email when KPI X’s value went up or down by a certain % or value. I put together a POC for them which uses SQL Integration Services (SSIS) to do the work. Note that this isn’t necessarily a best practice since as far as I know Microsoft doesn’t yet have any real guidance around how to replace SQLNS.

There were actually a bunch of different ways we could have approached this:

· Biztalk – rejected because it takes a while to grow BTS expertise and we’d be hitting a small-ish problem with an awfully big hammer

· Managed code and/or services which took care of alerting from soup-to-nuts – rejected because we didn’t want to recreate the wheel

· Managed code and/or services which managed part of the alerting process (detecting a change and dropping an event in the Windows Event log or raising a WMI event) and MOM or other operational management software to do the actual heavy lifting. We’d rely on MOM to see the newly added event and send whatever notice to whatever endpoint we specified. We rejected this because we couldn’t count on everyone having MOM running.

That pretty much left us with SSIS, which could do much of the work for us.

Here’s what this simple POC does:

· Collects “alert subscriptions” in a SQL table. Each subscription includes the email address to which we’ll send an alert, the KPI name in adventure works we’re monitoring, the value of that KPI when the subscription was created, and the “change factor” (up or down X%) which will cause the alert to fire

· Uses an SSIS Ipackage which reads each alert subscription and:

o Grabs the current value for the KPI we’re dealing with in the subscription currently being processed

o Compares the current value of the KPI to the “historical” one stored in the subscription table and sees if we’ve hit our change factor or not

o Sends an email to the user if we have

The POC is rough and makes no attempt at intelligence (grin). For example, we’ll happily send the same alert to a user over and over again when the package is run after we’ve hit the “change factor” for the subscription in question. Extra logic needs to be added to improve real-world usability of this thing.

Here are the components:

· A table which holds subscriptions:

USE [Catcher]

/****** Object: Table [dbo].[kpiDemo] Script Date: 10/22/2007 09:24:53 ******/

SET ANSI_NULLS ON

SET QUOTED_IDENTIFIER ON

CREATE TABLE [dbo].[kpiDemo](

[emailAddress] [nvarchar](50) NOT NULL,

[kpiName] [nvarchar](50) NOT NULL,

[kpiCurrentValue] [float] NOT NULL,

[kpiMovement] [float] NOT NULL,

CONSTRAINT [PK_kpiDemo] PRIMARY KEY CLUSTERED

(

[emailAddress] ASC

)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]

) ON [PRIMARY]

INSERT KPIDemo VALUES (‘someEmailAddress@domain.com’, ‘Channel Revenue’, 80000000, -0.1) -- Alert someEmailAddress when the Channel Revenue KPI value in AdventureWorks (with a current value of 80M) declines by 10%

· The package itself (attached below)

I had a bear of a time successfully reading KPIStatus, KPIValue, and KPIGoal values in the package. In essence, when I constructed an MDX query and fired it from an OLEDB Data Source or Data Reader, I’d get the following error message:

The output "OLE DB Source Output" (11) references an external data type that cannot be mapped to a Data Flow task data type. The Data Flow task data type DT_WSTR

I’d find that only two of the three columns I asked for came back, as well. The third (usually KPIStatus) would disappear into the ether. This made me sad and evidently is a known issue. I found a thread about it here but discovered the workaround did not help me:

http://forums.microsoft.com/TechNet/ShowPost.aspx?PostID=215075&SiteID=17

I ended up going with a Script Component acting as a source, and used code stolen from this great blog entry to get my values back:

http://blogs.msdn.com/olivier_pieri/archive/2004/12/06/275811.aspx

All standard disclaimers apply – use this at your own risk, it is unsupported and could burn down your house if left running unsupervised.

Dell and Microsoft engineer BI offering

russch — Thu, 20 Sep 2007 20:22:00 GMT

Today at SQL PASS, Dell and Microsoft announced a new set of Business Intelligence and Data Warehousing solutions. They include a stack of server hardware, storage arrays, operating system, database and analytics software, management and reporting tools. The entire solution is based on the hardware and software many customers are using right now.

What is really cool about these offerings is they answer the oft-asked question: “what sort of hardware do I need for my DW/BI project?”: Three reference configurations have been created: one terabyte, two terabyte and four terabyte systems for data warehousing databases using SQL Server 2005. Dell and Microsoft worked together to execute various tests and analyze performance results of a sample business intelligence and data warehousing system under load. The testing was designed to validate the platforms for business intelligence and data warehousing, to provide the stability and performance of the hardware systems, and to define the reference configurations I mentioned earlier.

From what I understand, we’ll essentially be including all the goodness in the SQL 2005 BI stack + ProClarity. Now that PerformancePoint 2007 is upon us, the offering will be reconfigured to include PPS at some point in the near future.

More official details should be available at ~~www.dell.com/bi~~ http://www.dell.com/sqlbi a little later today. It doesn't seem like the site is up yet, however.

Update:

Here is a link to the actual 1/2/4 TB reference configuration to see how they did it:

http://www.dell.com/content/topics/global.aspx/sitelets/solutions/software/db/microsoft_sql_2005_pr?c=us&cs=555&l=en&s=biz

...and another to an article on this event:

http://informationweek.com/news/showArticle.jhtml?articleID=201808018

SQL 2005 SP2 most likely available this Monday, 2/19

russch — Fri, 16 Feb 2007 17:35:00 GMT

Get ready to download, because SP 2 was declared "ready to ship" last night. The SQL team is hoping to have it ready for public download Monday-morning-ish (PST).

Reporting Services Login dialog appears when deploying to MOSS/WSS

russch — Wed, 07 Feb 2007 15:29:00 GMT

Brian Welcker does it again! In this post he describes a situation that's been causing me a great deal of angst. Essentially, after getting MOSS/SSRS integration working, you get prompted to login when publishing from BIDS - and entering credentials doesn't get you past the dialog, either.

I'm not big on "link wrangling", but over the past few days there have been too many good ones to pass up. Read this. Now.

Using a parameterized SSIS package as a data source for SQL Reporting Services

russch — Wed, 19 Jul 2006 22:34:00 GMT

As you may know, SSRS reports can use Integration Services packages as a source of data. This technique opens up some interesting possibilities around pre-processing data before it is rendered in a report.

I’m going to take this one step further and describe how to pass the package parameter values selected by the user in Reporting Services. These values are passed to the package so that it can do additional work like filter the resultset it returns to SSRS. You could also use this technique to send instructions to control of flow logic you have inside the package, etc.

The Basics:

First, if you haven’t played with SSIS as a report server data source, read these:

http://msdn2.microsoft.com/en-us/library/ms159215.aspx

http://msdn2.microsoft.com/en-us/library/ms156302.aspx

http://msdn2.microsoft.com/en-us/library/ms345250.aspx

After you have mastered the basics, follow the steps below…

Create an SSIS Package:

1. Create a new SSIS package, right-click the Control Flow tab, and choose Variables from the context menu.

2. Add a new variable: Name it filterValue, give it a Data Type of Int16 and a default value of 0.

3. Insert a single Data Flow Task into your control flow.

4. Drop an OLE DB Source in the data flow and configure it to connect to the AdventureWorks database.

5. Choose a Data access mode of SQL Command.

6. Next, we’re going to add a command which is parameterized. Enter or paste this text in the SQL Command text box:

SELECT     EmployeeID, NationalIDNumber, ContactID, LoginID, ManagerID, Title, BirthDate, MaritalStatus, Gender, HireDate, SalariedFlag, VacationHours,SickLeaveHours, CurrentFlag, rowguid, ModifiedDate
FROM     HumanResources.Employee
WHERE     (EmployeeID = ?)

7. Click the Parameters button. Add a Parameter Mapping of Parameter = EmployeeID, Variables = User::filterValue. Close the Set Query Parameters dialog, and you’ve basically created a simple SELECT which is dynamically filtered by whatever value lives in the filterValue variable of your package. (Note: Clicking Preview won’t work right now – no worries!)

8. Add a Data Reader Destination to your data flow and connect the OLE DB source to the Data Reader Destination.

9. Double-click the DataReaderDest you just dropped in the flow and choose the Input Columns tab. Select all the available input columns, and click OK.

10. From the Debug menu, choose Start Debugging to run your package. We won’t see anything especially interesting happen at this point, but we shouldn’t see any errors. Remember to Stop Debugging, then save and close your package.

11. I’d go ahead and copy the resulting *.dtsx to somewhere easy to get to, like the root of C:\

Create the report:

Create a new Report Server Project, right click the Reports folder in Solution Explorer, and add a New Item…In this case we’ll be creating a report.
Add a new Parameter to this report. Name it filterValue, set its data type as Integer, give it available values of 0, 1, 2, and 3, and finally set the default value of this parameter to 0.
Create a new Dataset in this report, and make sure to choose SSIS in the Type combo-box.
The magic happens in the Connection String box. Remember, that in SSRS 2005 we can actually build the connection string out of an expression. Our expression will call the SSIS package and dynamically change the value we pass in to the filterValue variable in the SSIS package:

= "/file c:\package.dtsx /Set \Package.Variables[filterValue];" & Parameters!filterValue.Value

5. Click OK. In the Query String box of the Dataset dialog, enter the name of the Data Reader Destination object in the SSIS package (probably “DataReaderDest”) .

6. That’s pretty much it. Go ahead and create the report as you normally would from the data that comes back. When you run the report you’ll be able to change the value for the filterList parameter, which is turn is passed to the SSIS package, and causes SQL to send you a filtered resultset via the Data Reader in the package.

SQL Server Integration Services: Sometimes BCP is "good enough".

russch — Sat, 06 Aug 2005 04:08:00 GMT

Today I sat in on a SSIS performance tuning "chalk talk" given by Ashvini Sharma at TechReady, an internal technical education event. Ashvini mentioned that when loading relatively small amounts of data into SQL (he characterized small as < 1 GB), you might be better served using "good enough" BCP.

Essentially, while the pipeline in SSIS is super-fast, it does do some up-front work before it starts copying rows that BCP does not. For any given dataset, BCP may therefore get a long enough head start at loading rows that SSIS and the pipeline can't catch up. Interesting, but I think SSIS is so cool I'd probably use it anyway...I hate typing those god-awful BCP command-lines.