Jamie's Junk : Code

Support Vector Machines for SQL Server Data Mining

JamieMac — Tue, 14 Oct 2008 20:43:17 GMT

Many have requested that we implement Support Vector Machines (SVM's) for SQL Server 2008, and for a wide variety of reasons, we just couldn't get to it. Luckily the community has come to the rescue for those needing an SVM implementation today! Joris Valkonet of Avanade Netherlands along with colleague Thanh Luc have implemented an SVM plug-in algorithm and viewer. Not only that, but Joris has released the plug-in along with all of the source code at CodePlex so you can customize the algorithm for your own purposes or at least get another example of how algorithms and viewers are implemented.

Below is a screenshot from the viewer showing cancer classification split across two selectable axes with green and blue indicating correctly classified benign and malignant tumors respectively and red indicating misclassifications.

The plug-in and code can be found at http://www.codeplex.com/svmplugin

Visual Numerics integration into SQL Server Data Mining

JamieMac — Fri, 30 May 2008 00:24:18 GMT

Our friends at Visual Numerics, Inc have created an excellent white paper covering the creation of SQL Server Data Mining plug-in algorithms in C#. In addition to documenting all of the architecture and concepts required, they have created a step-by-step tutorial demonstrating how to integrate their own K-Means clustering algorithm into SQL Server Data Mining using the C# interfaces. The entire data mining team here in SQL Server reviewed the paper - we couldn't have done a better job ourselves! Kudos to VNI for such an accomplishment!

The paper includes FULL SOURCE CODE for their plug-in algorithm and is available at http://www.vni.com/company/whitepapers/MicrosoftBIwithNumericalLibraries.pdf. Of course you will need to get a copy of their IMSL C# Numerical Library, for which an eval copy is available upon request to VNI.

As a teaser, here is the TOC for their paper:

Audience ..................................................................................................... 4
Rationale ..................................................................................................... 4
Background ................................................................................................. 5
Plug‐in Architecture .................................................................................... 8
Managed Plug‐in Development .................................................................. 9
IMSL C# Library: ClusterKMeans Integration........................................... 9
Starting up.............................................................................................. 10
Metadata Changes (Metadata.cs) ......................................................... 10
Algorithm Changes (Algorithm.cs) ......................................................... 11
Training and Persistence of patterns..................................................... 11
Persistence of Patterns .......................................................................... 13
Prediction............................................................................................... 13
Algorithm Navigator Changes (AlgorithmNavigator.cs) ........................ 13
Registering the Algorithm with Analysis Services.................................. 14
Debugging .............................................................................................. 15
Other Default Features for Third‐Party Mining Algorithm Developers.... 16
The User Experience ................................................................................. 16
Excel 2007 .............................................................................................. 19
Conclusion................................................................................................. 21
About the Author...................................................................................... 21
References ................................................................................................ 22
Appendix A: Code Files ............................................................................. 23

Tree Utilities in Analysis Services Stored Procedures

JamieMac — Fri, 02 Mar 2007 11:32:00 GMT

This past week I was helping out a customer that wanted to reduce the length of a questionairre by using data mining to determine which of the 300+ questions were actually necessary for them to get the understanding they required. By using a tree model, and playing with the COMPLEXITY_PENALTY parameter, I was able to build a model that was reasonably accurate and only required 10-17 questions. (I was able to do the whole project using the Data Mining Add-ins as well!)

In the process, I created some stored procedures that helped me easily extract the information I needed from the tree - information such as which paths are the shortest, the longest, that contain the minimum or maximum values, etc

Attached is the source code for those utilities. You can copy and paste them into Visual Studio - add a reference to ADOMDServer.NET, build and add as an assembly to your Analysis Services server.

The utilities give you the following functions:

For all trees:

List out the shortest paths from root to leaf
CALL TreeUtils.ShortestPaths("Model Name", "Tree Name")

List out the longest paths from root to leaf
CALL TreeUtils.LongestPaths("Model Name", "Tree Name")

Provide information about the shortest paths (e.g. value, probability, etc)
CALL TreeUtils.ShortestPathStatistics("Model Name", "Tree Name")

Provide information about the longest paths (e.g. value, probability, etc)
CALL TreeUtils.LongestPathStatistics("Model Name", "Tree Name")

For Regression Trees:

Return the path to the leaf node containing the minimum value
CALL TreeUtils.MinimumPath("Model Name", "Tree Name")

Return information about the path containing the minimum value (e.g. depth, value, etc)
CALL TreeUtils.MinimumPathStatistics("Model Name", "Tree Name")

Return the path to the leaf node containing the maxmum value
CALL TreeUtils.MaximumPath("Model Name", "Tree Name")

Return information about the path containing the minimum value (e.g. depth, value, etc)
CALL TreeUtils.MaximumPathStatistics("Model Name", "Tree Name")

For Classification Trees

Return the path leading to the leaf with the least likelihood of the specified state
CALL TreeUtils.MinimumPath("Model Name", "Tree Name", "State")

Return the path leading to the leaf with the most likelihood of the specified state
CALL TreeUtils.MaximumPath("Model Name", "Tree Name", "State")

Return information about the path containing the least likelihood of the state (e.g. depth, probability, etc)
CALL TreeUtils.MinimumPathStatistics("Model Name", "Tree Name", "State")

Return information about the path containing the most likelihood of the state (e.g. depth, probability, etc)
CALL TreeUtils.MaximumPathStatistics("Model Name", "Tree Name", "State")

Social Network Analysis with SQL Server 2005 Data Mining

JamieMac — Tue, 27 Feb 2007 02:34:00 GMT

A few years ago John Hancock spent some time with our team and did some interesting work around using the Association Rules algorithm and some custom Analysis Services stored procedures to implement link analysis (a.k.a. social network analysis). His implementaion was extremely novel, and the product team (self included) was duly impressed by what John was able to eek out of our system.

Well, as he puts it himself he "finally got around to posting" all the source, etc. for this work for the enlightenment of everyone!

I'm happy to hear of (and promote!) any other slick projects using SSDM that you want to share!

Automatically Label Clusters using Analysis Services Stored Procedures

JamieMac — Thu, 22 Feb 2007 01:06:00 GMT

I just found this awesome posting on CodePlex by furmangg to automatically label the clusters generated by the clustering algorithm. It examines the cluster content to generate human-readable descriptions and then changes the labels appropriately. Check it out - it might just save you a bunch of time figuring out what all the clusters mean.

Direct Links to Data Mining feature pack items

JamieMac — Wed, 26 Jul 2006 20:54:00 GMT

Many people have asked me for the links to the embeddable viewers and the .NET interfaces for plug-in algorithms. Instead of my usual response, "go to the feature pack page and scroll until you find them", here are the direct links. Enjoy!

Data Mining Embeddable Viewers
http://www.microsoft.com/downloads/details.aspx?FamilyID=df0ba5aa-b4bd-4705-aa0a-b477ba72a9cb#dmviewer

Managed Plug-in API
http://www.microsoft.com/downloads/details.aspx?familyid=df0ba5aa-b4bd-4705-aa0a-b477ba72a9cb#dmapi

Also, you may need to download the OLEDB and ADOMD.NET providers as well for client applications

ADOMD.NET
http://www.microsoft.com/downloads/details.aspx?FamilyID=df0ba5aa-b4bd-4705-aa0a-b477ba72a9cb#adomdnet

OLEDB Provider
http://www.microsoft.com/downloads/details.aspx?FamilyID=df0ba5aa-b4bd-4705-aa0a-b477ba72a9cb#asoledb

Managed Plug-ins at Last!

JamieMac — Thu, 20 Apr 2006 01:34:00 GMT

For all those of you who have been waiting and waiting to implement your own data mining algorithms for SQL Server 2005, but didn't want to get into the complexity of writing C++ COM code - it's finally here! The Managed Plug-in API's are available as part of the SQL Server 2005 feature pack. Scroll all the way to the bottom - you'll find it there.

Included with the package are

All the managed interop code
A complete help file with conceptual and reference information
A tutorial to help you get started

In addition to wrapping up all of the plug-in API's nicely, Bogdan added a bunch of cool ways to do things like add your own mining functions, parameters, and notifications.

Enjoy! And thanks to Bogdan for all of his work in designing, implementing, and documenting these API's!

Automatic Generation of CREATE MINING MODEL statements

JamieMac — Thu, 30 Mar 2006 21:51:00 GMT

I noticed through answering newsgroup and forum posts that we always want to know model definitions when discussing a problem. Luckily we have this great language called DMX that makes it very easy and simple to describe a model. Unluckily, our tools that we use to create models use an XML DDL that is very complex and, for the most part, not intelligible to human readers.

To solve this problem I sat down last night and wrote up a little stored procedure to create a CREATE MINING MODEL statement from any server model. All you have to do after building and deploying the stored procedure is execute the query

SELECT DMXtract.CreateMiningModelStatement() FROM [TreeModel]

To build this code you will have to add a reference to Microsoft.AnalysisServices.AdomdServer. For more information on how to deploy stored procedures see this article in Books Online.

Below is the code for CreateMiningModelStatement. One unforeseen benefit I discovered writing this code was that the parameters collection contains all parameters even if they weren’t set by the user. This makes it useful to determine how the algorithms automatically set specific parameter values on your behalf.

using System;

using System.Collections.Generic;

using System.Text;

using Microsoft.AnalysisServices.AdomdServer;

namespace DMXtract

{

public class DMXtract

{

[SafeToPrepare(true)]

public string CreateMiningModelStatement()

{

// Return empty string on prepare

if (Context.ExecuteForPrepare)

return "";

// Return the create statement for the current model

return CreateMiningModelStatement(Context.CurrentMiningModel);

}

private string CreateMiningModelStatement(MiningModel model)

{

int count = 0;

// Add the CREATE syntax

string statement = "CREATE MINING MODEL [" + model.Name + "]";

statement += "\r\n(\r\n";

// Add the columns

statement += ListModelColumns(model.Columns,1);

// Add the algorithm

statement += "\r\n) USING ";

statement += model.Algorithm;

// Add model parameters

if (model.Parameters.Count > 0)

{

statement += "(";

foreach(MiningParameter param in model.Parameters)

{

if (param.Name == "" | param.Value == "")

continue;

if (count > 0)

statement += ", ";

statement += param.Name + "=" + param.Value;

count++;

}

statement += ")";

}

// Add Drillthrough

if (model.AllowDrillThrough)

statement += "\r\nWITH DRILLTHROUGH";

return statement;

}

private string ListModelColumns(MiningModelColumnCollection columns, int indent)

{

int count = 0;

string statement = "";

// Iterate columns in collection

foreach (MiningModelColumn modelcol in columns)

{

if (count > 0)

statement = statement + ",\r\n";

// Indent as indicated

for (int i = 0; i < indent; i++)

statement += " ";

statement += "[" + modelcol.Name + "]\t";

MiningStructureColumn structcol = modelcol.StructureColumn;

if (modelcol.Type != MiningColumnType.Table)

{

// Append scalar column

statement += " " + modelcol.Type.ToString().ToUpper();

if (modelcol.Distribution != MiningColumnDistribution.Missing)

statement += " " + modelcol.Distribution.ToString().ToUpper();

if (modelcol.Flags != "")

statement += " " + modelcol.Flags;

statement += " " + modelcol.Content.ToString().ToUpper();

// Add prediction flags

if (modelcol.IsPredictable)

{

if (modelcol.IsInput)

statement += " PREDICT";

else

statement += " PREDICT_ONLY";

}

else

{

// Append nested table column

statement += " TABLE";

// Add prediction flags

if (modelcol.IsPredictable)

{

if (modelcol.IsInput)

statement += " PREDICT";

else

statement += " PREDICT_ONLY";

}

statement += "\r\n";

for (int i = 0; i < indent; i++)

statement += " ";

statement += "(\r\n";

// Append nested columns

statement += ListModelColumns(modelcol.Columns, indent + 1);

// Close nested table definition

statement += "\r\n";

for (int i = 0; i < indent; i++)

statement += " ";

statement += ")";

}

count++;

}

return statement;

}

Two similar but different resources now available

JamieMac — Thu, 08 Dec 2005 22:39:00 GMT

There are two new resources available for writing custom algorithms for SQL Server Data Mining

"Add Custom Data Mining Algorithms to SQL Server 2005" by our own Raman Iyer, provides an overview of the plug-in process, what you do and what you get, along with links to more technical resources.

Implementing Data Mining Algorithms in Microsoft® SQL Server™ by Claudio Luiz Curotto is an in-depth text on implementing algorithms for SQL Server 2000.

Code from book and new XMLA template at SSDM

JamieMac — Tue, 15 Nov 2005 23:07:00 GMT

I just added a couple of new downloads on SQLServerDataMining.com

The first is the sample from chapter 16 (Forecasting in Excel) of my book, Data Mining with SQL Server 2005 which was misposted on the publisher's site (it will be updated there as well, but it's at SSDM for those of you who want it today).

The other download is a simple XMLA template that is surprisingly important since it fills a gap in our DMX language that was blocking common workflows.

DMX has great constructs for creating, training, and managing mining structures and models. Now that SQL Server Management Studio (SSMS) has a DMX editor, it's really nice to just open one up and start typing CREATE MINING STRUCTURE ... and do all of your mining right there. However, in general, once you got to INSERT INTO you were stuck (not to mention PREDICTION JOIN). Since we strongly encourage everyone to use OPENQUERY and a named datasource rather than OPENROWSET and a connection string, it would seem that there would be an easy way to create datasources in SSMS. Wrong.

The only "easy" way to create a datasource would be to fire up the BI Dev Studio, open the database, create a datasource in the UI and then switch back to SSMS - a user experience that leaves a bad taste in your mouth - essentially a big hole in the experience.

So the template I added allows you to easily create a datasource from SSMS by loading the template, setting some values (like a name, connection string, etc) and executing. Now you can perform your entire DM workflow directly in SSMS!

Enjoy! I welcome any feedback or many suggestions for new templates.

Updated sample viewer for Sept CTP and RTM

JamieMac — Sat, 22 Oct 2005 02:15:00 GMT

A couple people found that the sample plug-in viewer I posted on SQLServerDataMining.com didn't install on the September CTP or later builds (although it continued to work if already installed). I posted a new version that installs correctly with the newer builds.

If you want it soon, grab it right away, we're taking the site down on Monday to upgrade to the latest RC bits.

On PASS, PDC and new Downloadable Demos

JamieMac — Sat, 01 Oct 2005 00:24:00 GMT

I've just finished uploading some new demos that I've shown at PDC and PASS. Both of these demos show how to use VBA to integrate data mining functionality into Excel. Their pretty cool - you should check them out. Bogdan wrote the DMPredict function and I wrote the clustering sample.

As I'm writing this PASS is finishing up - I actually should be on the way to the airport, but I still have a couple minutes....

PASS was excellent as usual and I'm fired up about the response to the data mining tools in SQL 2005 - there's a whole new audience out there that has advanced research technology available to them for the first time. It will be exciting to see how these communities can push and pull this tech to do things that no one has ever imagined. A PDC'er that commented to be two weeks back in LA, called the advances in Data Mining and OLAP in SQL Server 2005 Analysis Services a "seismic" event. It's good to be in AS!

Go ahead - if you've downloaded the September CTP, check out those demos! Gotta catch my flight!

Updated viewer client sample for the June CTP

JamieMac — Sat, 16 Jul 2005 21:03:00 GMT

I updated the DataMiningViewerClient sample on the downloads section of www.sqlserverdatamining.com. This sample demonstrates how to host the data mining viewers from SQL Server 2005 in your own applications, and allows you to view models side-by-side. For the most part, the update involved removing certain "workarounds" that were required for the beta 2 version. The download includes the working application plus source code.

New data mining sample available

JamieMac — Thu, 14 Jul 2005 00:41:00 GMT

Those of you who read my blog likely know that our site went down again (not because I said so, but because there's a high correlation between people who read my blog and visit sqlserverdatamining.com) This time the reason was a bit more benign - in moving the box to it's new location, the hard drive shook loose and needed to be reseated. In the process, we added a second drive and ordered some more memory so we won't have these issues anymore - or as much, at least.

Anyway to commemorate the occasion of the site's rebirth I added a new sample to our downloads section - this time a sample plug-in viewer that provides a generic method of browsing the content of any model - regardless of the algorithm. It's a great tool for those who want to understand the content layout of the algorithms so you can write your own applications - or for those writing their own algorithms that need to debug how you are generating your content. Also, the source code is included for those who want to create your own plug-in viewers as well.

Along the same topic, if you missed my programming webcast, Intelligent Applications: Embedding Data Mining in Your Application , it is now available on-demand.

Hope you enjoy it! Keep on mining....

Text Mining and AMO in MSDN

JamieMac — Tue, 10 May 2005 22:19:00 GMT

The current issue of MSDN has two data mining articles

The first Say What? Make Sense of Your Web Feedback using SQL Server 2005 by Peter Kim takes you through the steps of using SQL Server 2k5 Text Mining and Data Mining to extract information from verbatim comments.

The second Mine Your Business: AMO Lets You Dig Deeper into Your Data from Your Own Applications by our own Liu Tang and Paul Bradley from Apollo Data Tech gives some insight on why you would add data mining processing to your applications and shows you how to do it.