Jamie's Junk : Decision Trees

To (a) or not to (a), that is the question?

JamieMac — Mon, 14 Jan 2008 08:36:00 GMT

While looking for content for the next edition of my book (newsflash! I'm currently working on the next edition of my book!) I went hunting around for that trick using decision trees to predict the states of a single column independently rather than all together. Turns out - I never wrote it! So, in case you don' t want to wait for it (or it doesn't make it into the book!), here it is now.

The classifier-type algorithms in SQL Server Data Mining (notably decision trees, naive bayes, neural networks) can all predict multinomial outputs - that is, output attributes with multiple states - 3,4, 10, 20, whatever. However, in reality, classifiers in general prefer the ying and yang of things, the black and white, the yes-ness and the no-ness. In short, they really are better at seperating between the states of a binomial attribute rather than those of a multinomial attribute - and so are you, actually. If I gave you ten things to look at and said what is the factor that most cleanly divides all ten of these things, you'd have a hard time, but if I gave you two things instead, you might not have a problem. You would be more accurate, and your model may be more accurate as well, if you only had binomials.

Additionally, with multinomials, your model - particularly with decision trees - is harder to interpret. Say you have the marital status states of "Married", "Single", "Seperated", "Divorced" and "Widowed". When I look at the dependency net and it shows that "Number of Children" is predictive of "Marital Status" - what does it mean? Which aspect of marital status is it talking about - all of them? One of them? Impossible to tell.

So what do you do when you have a multinomial output? The first choice is obvious - see if you can reduce it to two states - e.g. can they be changed into "Married" and "Not Married". If that's not an option, i.e. if the states are important, another option is to turn them into a series of binomials - e.g. "Marriend" and "Not Married", "Single" and "Not Single", etc.

Now we run into an additional problem - not only is transposing our data like that a royal pain in the butt, it increases the number of columns and we may exceed the maximum row length of SQL Server.

Luckily with SQL Server Data Mining, we have a way out. By using a trick with nested tables, we can create a model that treats each state as a binomial attribute without changing any of our data. Assume our original model looked like this:

CREATE MINING MODEL MyMultinomialModel
(
    CustomerID     LONG   KEY,
    Age            LONG   CONTINUOUS,
    Gender         TEXT   DISCRETE,
    NumChildren   LONG   CONTINUOUS,
    MaritalStatus TEXT   DISCRETE     PREDICT
) USING Microsoft_Decision_Trees

we can transform the MaritalStatus field into a nested table like this:

CREATE MINING MODEL MyBinomialModel
(
    CustomerID     LONG   KEY,
    Age            LONG   CONTINUOUS,
    Gender         TEXT   DISCRETE,
    NumChildren   LONG   CONTINUOUS,
    MaritalStatus TABLE PREDICT ONLY
    (
         Status    TEXT   KEY
    )
) USING Microsoft_Decision_Trees

OK, so now that we've created the model, how do we train it - we only have a single source table right? Well, you're right, but that doesn't really matter. Using the tools in BI Dev Studio, you can select a table as both a Case and Nested table, and you can do the same thing using DMX, like this

INSERT INTO MyBinomialModel(CustomerID,
                            Age, Gender, NumChildren,
                            MaritalStatus(SKIP, Status))
SHAPE{OPENQUERY(MyDataSource,
                'SELECT CustID, Age, Gender, NumChildren
                FROM Customers ORDER BY CustID') }
      APPEND
      {OPENQUERY(MyDataSource,'SELECT CustID, MaritalStatus
                              FROM Customers ORDER BY CustID')
      RELATE CustID to CustID} AS MaritalStatus

So why does this work? Since we're using the same table as the case and nested tables, we're guaranteed that each case will have one and only one "Status" in the "MaritalStatus" table. The MaritalStatus table is PREDICT ONLY, so there's no cross-confusion between the states. A decision tree model in this case will build five trees - one for each state of MaritalStatus. Each attribute is a binomial variable with two possibilities "This state exists" or "This state doesn't exist".

This also helps in prediction if you need to predict a particular state. You will get a much better prediction of whether or not a customer is or is not a particular state than if you did this the traditional way. You can find out how likely by using subselects with your prediction statements, e.g.

SELECT FLATTENED
   (SELECT * FROM
      PREDICT(MaritalStatus, INCLUDE_STATISTICS)
    WHERE Status='Single')
FROM MyBinomialModel PREDICTION JOIN ...

The above query will give you the probability and support of being single for each customer in the input set.

Additionally, in the dependency network, each state of MaritalStatus will appear as its own node, so you will finally be able to see whether Number Of Children is more predictive of "Married" or of "Divorced" J

And for those wanting more information on the book, I’m working once again with Zhaohui Tang on the book and this time we’ve added Bogdan Crivat as a new author. It will probably have some spiffy name like “Data Mining with SQL Server 2008” and will be released sometime around when the product ships – no comment on that!

(note: the author is happily married with four children, regardless of what your dependency net says)

How can we mine? Let me count the ways...

JamieMac — Tue, 20 Nov 2007 00:18:10 GMT

Recently I received some customer feedback that SQL Server Data Mining "doesn't have enough algorithms." More specifically, the comment was that we have the same capabilities are other Data Mining providers, we just "hide" many facilities as algorithm parameters rather than separating out each as a named algorithm. So let's count the Microsoft algorithms a few different ways to work this out.

First - let's go by the box. This is the list of algorithms as specified in Books Online

Microsoft Decision Trees
Microsoft Clustering
Microsoft Naive Bayes
Microsoft Association Rules
Microsoft Neural Networks
Microsoft Time Series
Microsoft Sequence Clustering
Microsoft Linear Regression
Microsoft Logistic Regression

So that's nine - count 'em nine algorithms. But that's just one way. If you look at my book, Data Mining with SQL Server 2005 written with Zhaohui Tang, there are only seven algorithms! What? You say! How can it be?

Let me explain. During the development of SQL Server 2005, we realized a couple of tricks; 1) linear regression was the same as our tree algorithm, just forced to not split; and 2) logistic regression was the same as our Neural Nets, just with zero hidden layers. However, we got similar feedback - people want more algorithms, and specifically these ones, so we set up two "new algorithms" by forcibly setting parameters on the Decision Tree and Neural Network algorithms and voila! we shipped with nine named algorithms. It would have been hard to fill up two entire chapters explaining that last sentence, so Zhaohui and I decided just to stick to the seven core algorithms.

Anyway, this posting isn't really about how to count less algorithms, I really wanted to show you how to count more. When we set about designing SQL Server Data Mining, we really and truly tried to make data mining operations simpler. We thought at the time, rightly or wrongly, that the more options end users have, the more complicated and difficult the product would be to use. Therefore, we tried to determine the best behavior in a class, and make more advanced options available through parameters.

For example, take our clustering algorithm. We assumed that if people wanted clustering, most likely didn't care about the details of the algorithm, they just wanted to get the job done, and that those people who wanted more would look for it (the design principal - make the simple things simple, and the complex things possible). So we bundled up different flavors of clustering into a single package that many vendors would have broken apart. So let's start counting with clustering.

Our default clustering behavior is EM (Expectation Maximization) clustering using the Bradley-Fayyad scalable framework

Setting a parameter changes that to a K-Means clustering implementation using the same framework

3+4

Setting the same parameter another way, provides non-scalable versions of the two clustering varieties. (I know it's hard to swallow that the non-scalable versions count as separate algorithms, but if you started with the vanilla versions and added scalability, then of course you would consider those versions as new algorithms - I'm just working backwards here.

Let's move to our Decision Tree algorithm and we will consider our classification tree as one algorithm.

But our Decision Tree also predicts continuous and counts as a regression tree, so we will count that as another algorithm.

Oops! Our Decision Tree also creates full linear regressions at each of the leaf nodes. To get the typical regression tree behavior you need to make sure that none of the continuous inputs have the REGRESSOR flag and you get yet another algorithm.

Oh yeah, our trees allow for multiple targets in each model, allowing the discovery and display of interrelated patterns through our dependency net. I've seen other vendors advertise such functionality as an "algorithm" so there's our #8.

How about collaborative filtering with Trees - just slap a PREDICT flag on a nested table, and you have a complete recommendation system. Let's call it Associative Trees

Naive Bayes.

11+12

If we're going to count Associative Trees, we also have "Associative Bayes". I guess the multiple target interrelated pattern thing counts here as well.

Association Rules. A-priori style

It seems odd to count association rules twice since we can do predictions with it, but nobody else does it (or didn't before - correct me if I'm wrong), so Predictive Association Rules makes the cut.

15+16+17+18

Well if we're going to go and call predictive association an algorithm, we had better do the same for our clustering algorithm. Granted, clustering doesn't make a great classifier or estimator, but the great Highlight Exceptions functionality of the Data Mining addins comes from this ability. Yes, we can do nested table prediction as well with clustering, but I wouldn't recommend it to my mom, so I won't take another four for that.

19+20+21+22+23

Neural Networks, Sequence Clustering, Time Series, Linear Regression and Logistic Regression. Yeah, yeah, I could get into varieties here, but I think you get the point.

So by that count, and not being too creative (trust me, I can do more) we're looking at 23 algorithms in SQL Server 2005 Data Mining. There are a few more options coming up in SQL Server 2008 that are worth discussing as well.

The time series of SQL Server 2007 uses the ARTXP algorithm - "Auto Regression Trees with Cross Predict". In 2008, we're adding ARIMA as well, for algorithm #24.

And yet again with Time Series, the default mode of operation is to blend ARTXP and ARIMA results in an intelligent way to maximize accuracy and stability for #25.

Arbitrarily there are 23 algorithms in SQL 2005 and 25 in SQL 2008, with the option of teasing out even more varieties depending on how you apply parameters and flags to the base nine (or seven - depending on how you count!). Next time someone quips that SQL Server only has "nine" algorithms, tell them that's just the packaging - each of those nine provides a wealth of value in each box.

Tree Utilities in Analysis Services Stored Procedures

JamieMac — Fri, 02 Mar 2007 11:32:00 GMT

This past week I was helping out a customer that wanted to reduce the length of a questionairre by using data mining to determine which of the 300+ questions were actually necessary for them to get the understanding they required. By using a tree model, and playing with the COMPLEXITY_PENALTY parameter, I was able to build a model that was reasonably accurate and only required 10-17 questions. (I was able to do the whole project using the Data Mining Add-ins as well!)

In the process, I created some stored procedures that helped me easily extract the information I needed from the tree - information such as which paths are the shortest, the longest, that contain the minimum or maximum values, etc

Attached is the source code for those utilities. You can copy and paste them into Visual Studio - add a reference to ADOMDServer.NET, build and add as an assembly to your Analysis Services server.

The utilities give you the following functions:

For all trees:

List out the shortest paths from root to leaf
CALL TreeUtils.ShortestPaths("Model Name", "Tree Name")

List out the longest paths from root to leaf
CALL TreeUtils.LongestPaths("Model Name", "Tree Name")

Provide information about the shortest paths (e.g. value, probability, etc)
CALL TreeUtils.ShortestPathStatistics("Model Name", "Tree Name")

Provide information about the longest paths (e.g. value, probability, etc)
CALL TreeUtils.LongestPathStatistics("Model Name", "Tree Name")

For Regression Trees:

Return the path to the leaf node containing the minimum value
CALL TreeUtils.MinimumPath("Model Name", "Tree Name")

Return information about the path containing the minimum value (e.g. depth, value, etc)
CALL TreeUtils.MinimumPathStatistics("Model Name", "Tree Name")

Return the path to the leaf node containing the maxmum value
CALL TreeUtils.MaximumPath("Model Name", "Tree Name")

Return information about the path containing the minimum value (e.g. depth, value, etc)
CALL TreeUtils.MaximumPathStatistics("Model Name", "Tree Name")

For Classification Trees

Return the path leading to the leaf with the least likelihood of the specified state
CALL TreeUtils.MinimumPath("Model Name", "Tree Name", "State")

Return the path leading to the leaf with the most likelihood of the specified state
CALL TreeUtils.MaximumPath("Model Name", "Tree Name", "State")

Return information about the path containing the least likelihood of the state (e.g. depth, probability, etc)
CALL TreeUtils.MinimumPathStatistics("Model Name", "Tree Name", "State")

Return information about the path containing the most likelihood of the state (e.g. depth, probability, etc)
CALL TreeUtils.MaximumPathStatistics("Model Name", "Tree Name", "State")

Wisconsin Breast Cancer Dataset available

JamieMac — Fri, 02 Feb 2007 01:50:00 GMT

Frequently I use the Wisconsin Breast Cancer Dataset for demonstrating the Data Mining Addins for Office - enough people asked, so I made it available as an Excel 2007 file (free login required). For purists, the original data is available at the Machine Learning repository, which is a great location for many sample datasets.

Here are some screenshots of the data mining add-ins applied to this dataset

Figure 1: Key Factor Analysis showing differences between benign and malignant tumors

Figure 2: Detect categories showing malignancy across detected groups. Note two purely malignant categories suggesting differing classes of malignant tumors.

Figure 3: Decision tree to predict diagnosis, with nodes shaded based on likelihood of malignancy.