Jamie's Junk : Clustering

Getting Data Mining results into SQL Tables

JamieMac — Wed, 08 Oct 2008 09:35:00 GMT

I've been seeing lots of questions about how to get data mining results into SQL tables. Most times the answers are "use the prediction query builder save button" or "build an SSIS package." Both of these have issues among them being that the former is really only suited to single-use, small jobs, and the latter has a lot of overhead (not to mention that if you want to use the Data Mining Query Transform you have to have Enterprise Edition).

Luckily, there happens to be a much easier way - and it's one of those "Doh!" moments when you learn about it, because it's that easy.

The way to do it is to simply use linked servers. Anyone who uses DMX knows to connect to SQL data with OPENQUERY. There's no reason you can't simply connect to Data Mining data using the same mechanism.

For example, use a query like this to establish a link to an AS server:

EXEC sp_addlinkedserver 
@server='LINKED_AS', -- local SQL name given to the linked server
@srvproduct='', -- not used 
@provider='MSOLAP', -- OLE DB provider 
@datasrc='localhost', -- analysis server name (machine name) 
@catalog='MovieClick' -- default catalog/database

Then you can select data using OPENQUERY like this:

SELECT * FROM 
OPENQUERY(LINKED_AS, 
  'SELECT Cluster() AS [Cluster], ClusterProbability() AS [Prob] 
   FROM [Customers - Clustering]
   NATURAL PREDICTION JOIN
   OPENQUERY([Movie Click],''SELECT * FROM Customers'') AS t')

Then, of course, you can do all kinds of manipulations on it, like finding the average cluster probability of each cluster, right? Well, almost, the data type returned by the Cluster function is actual text or ntext or something that GROUP BY chokes on, so you have to do some casting first. Therefore if you want to do that trick, use a query like this:

SELECT Cluster, AVG(Prob) FROM
(SELECT CAST(Cluster AS Char(30)) AS Cluster, Prob FROM OPENQUERY(LINKED_AS, 
  'SELECT Cluster() AS [Cluster], 
      ClusterProbability() AS [Prob] FROM [Customers - Clustering]
   NATURAL PREDICTION JOIN
   OPENQUERY([Movie Click],''SELECT * FROM Customers'') AS t')
   ) AS t
GROUP BY Cluster

That will give you a nice result showing you, in a way, the affinity of each cluster based on the input set. That is, if you ran such a query against the training data, you could say that the clusters with a higher probability are "tighter" than the ones with low probabilities. Anyway, that's besides the point of this post.

In any case, remember to double your single quotes and flatten any nested results and this technique should work just great for getting DMX into SQL.

-J

How can we mine? Let me count the ways...

JamieMac — Tue, 20 Nov 2007 00:18:10 GMT

Recently I received some customer feedback that SQL Server Data Mining "doesn't have enough algorithms." More specifically, the comment was that we have the same capabilities are other Data Mining providers, we just "hide" many facilities as algorithm parameters rather than separating out each as a named algorithm. So let's count the Microsoft algorithms a few different ways to work this out.

First - let's go by the box. This is the list of algorithms as specified in Books Online

Microsoft Decision Trees
Microsoft Clustering
Microsoft Naive Bayes
Microsoft Association Rules
Microsoft Neural Networks
Microsoft Time Series
Microsoft Sequence Clustering
Microsoft Linear Regression
Microsoft Logistic Regression

So that's nine - count 'em nine algorithms. But that's just one way. If you look at my book, Data Mining with SQL Server 2005 written with Zhaohui Tang, there are only seven algorithms! What? You say! How can it be?

Let me explain. During the development of SQL Server 2005, we realized a couple of tricks; 1) linear regression was the same as our tree algorithm, just forced to not split; and 2) logistic regression was the same as our Neural Nets, just with zero hidden layers. However, we got similar feedback - people want more algorithms, and specifically these ones, so we set up two "new algorithms" by forcibly setting parameters on the Decision Tree and Neural Network algorithms and voila! we shipped with nine named algorithms. It would have been hard to fill up two entire chapters explaining that last sentence, so Zhaohui and I decided just to stick to the seven core algorithms.

Anyway, this posting isn't really about how to count less algorithms, I really wanted to show you how to count more. When we set about designing SQL Server Data Mining, we really and truly tried to make data mining operations simpler. We thought at the time, rightly or wrongly, that the more options end users have, the more complicated and difficult the product would be to use. Therefore, we tried to determine the best behavior in a class, and make more advanced options available through parameters.

For example, take our clustering algorithm. We assumed that if people wanted clustering, most likely didn't care about the details of the algorithm, they just wanted to get the job done, and that those people who wanted more would look for it (the design principal - make the simple things simple, and the complex things possible). So we bundled up different flavors of clustering into a single package that many vendors would have broken apart. So let's start counting with clustering.

Our default clustering behavior is EM (Expectation Maximization) clustering using the Bradley-Fayyad scalable framework

Setting a parameter changes that to a K-Means clustering implementation using the same framework

3+4

Setting the same parameter another way, provides non-scalable versions of the two clustering varieties. (I know it's hard to swallow that the non-scalable versions count as separate algorithms, but if you started with the vanilla versions and added scalability, then of course you would consider those versions as new algorithms - I'm just working backwards here.

Let's move to our Decision Tree algorithm and we will consider our classification tree as one algorithm.

But our Decision Tree also predicts continuous and counts as a regression tree, so we will count that as another algorithm.

Oops! Our Decision Tree also creates full linear regressions at each of the leaf nodes. To get the typical regression tree behavior you need to make sure that none of the continuous inputs have the REGRESSOR flag and you get yet another algorithm.

Oh yeah, our trees allow for multiple targets in each model, allowing the discovery and display of interrelated patterns through our dependency net. I've seen other vendors advertise such functionality as an "algorithm" so there's our #8.

How about collaborative filtering with Trees - just slap a PREDICT flag on a nested table, and you have a complete recommendation system. Let's call it Associative Trees

Naive Bayes.

11+12

If we're going to count Associative Trees, we also have "Associative Bayes". I guess the multiple target interrelated pattern thing counts here as well.

Association Rules. A-priori style

It seems odd to count association rules twice since we can do predictions with it, but nobody else does it (or didn't before - correct me if I'm wrong), so Predictive Association Rules makes the cut.

15+16+17+18

Well if we're going to go and call predictive association an algorithm, we had better do the same for our clustering algorithm. Granted, clustering doesn't make a great classifier or estimator, but the great Highlight Exceptions functionality of the Data Mining addins comes from this ability. Yes, we can do nested table prediction as well with clustering, but I wouldn't recommend it to my mom, so I won't take another four for that.

19+20+21+22+23

Neural Networks, Sequence Clustering, Time Series, Linear Regression and Logistic Regression. Yeah, yeah, I could get into varieties here, but I think you get the point.

So by that count, and not being too creative (trust me, I can do more) we're looking at 23 algorithms in SQL Server 2005 Data Mining. There are a few more options coming up in SQL Server 2008 that are worth discussing as well.

The time series of SQL Server 2007 uses the ARTXP algorithm - "Auto Regression Trees with Cross Predict". In 2008, we're adding ARIMA as well, for algorithm #24.

And yet again with Time Series, the default mode of operation is to blend ARTXP and ARIMA results in an intelligent way to maximize accuracy and stability for #25.

Arbitrarily there are 23 algorithms in SQL 2005 and 25 in SQL 2008, with the option of teasing out even more varieties depending on how you apply parameters and flags to the base nine (or seven - depending on how you count!). Next time someone quips that SQL Server only has "nine" algorithms, tell them that's just the packaging - each of those nine provides a wealth of value in each box.

Automatically Label Clusters using Analysis Services Stored Procedures

JamieMac — Thu, 22 Feb 2007 01:06:00 GMT

I just found this awesome posting on CodePlex by furmangg to automatically label the clusters generated by the clustering algorithm. It examines the cluster content to generate human-readable descriptions and then changes the labels appropriately. Check it out - it might just save you a bunch of time figuring out what all the clusters mean.

Wisconsin Breast Cancer Dataset available

JamieMac — Fri, 02 Feb 2007 01:50:00 GMT

Frequently I use the Wisconsin Breast Cancer Dataset for demonstrating the Data Mining Addins for Office - enough people asked, so I made it available as an Excel 2007 file (free login required). For purists, the original data is available at the Machine Learning repository, which is a great location for many sample datasets.

Here are some screenshots of the data mining add-ins applied to this dataset

Figure 1: Key Factor Analysis showing differences between benign and malignant tumors

Figure 2: Detect categories showing malignancy across detected groups. Note two purely malignant categories suggesting differing classes of malignant tumors.

Figure 3: Decision tree to predict diagnosis, with nodes shaded based on likelihood of malignancy.