Microsoft Bob

Now a SQL Master

Bob Leithiser — Sat, 21 Nov 2009 19:25:07 GMT

After a long and painful process, I have finally achieved this. Was it worth it, would I do it again? Absolutely. But please don’t call me Master Bob, Microsoft Bob is fine… I am doing quite a bit more work with my simulation database...(read more)

Trying to catch up

Bob Leithiser — Tue, 06 Oct 2009 17:59:00 GMT

I apologize as I notice there are some inquiries related to code posted. I should probably set expectations about this, as I am not the typical Microsoft employee with a MSDN blog. Most of the other folks with MSDN blogs are in the product teams and blogging is part of their job. With me, it is an extra activity and my real job is being committed 24x7 to a high profile customer which takes priority over all my other professional activities. I provide this blog as a way to share with the community, but there is no committment implied to fix any issues that people may find in the code or respond quickly to questions.

Depending on customer situation, I may have time to frequently post as I did in July/August posting about a dozen articles, othertimes like this last month, customer issues take priority and I ma not post at all.

I apologize for not being able to provide quicker turn-around on inquiries and will try to catch up with the questions/requests over the next week or two.

Thanks for your patience.

Data-mining the Web for Fun [and Profit??]

Bob Leithiser — Mon, 24 Aug 2009 23:48:02 GMT

I’m trying to get more interaction on my blog by posing more provocative titles... This weekend I downloaded the last 10 years of Financial fundamental data for virtually all of the major AMEX, NYSE, and NASDAQ stocks currently traded, which resulted in a data table containing about 60,000 rows for 7,000 different equities (some equities are less than 10 years old, so not all have 10 years of history). These are not historical stock quotes that you can download from Yahoo, but historical financial fundamentals of earnings, book value, return on equity. This type of information is invaluable for historical analysis to find correlations between financial metrics and stock performance. To get the quality and quantity of information in electronic format could costs several hundred dollars from a typical investment subscription service.

Curious? I’ll try to walk you through the process, but first an overview. There is an amazing amount of semi-structured data on the web, just waiting to be sucked into databases in order to enable advanced analytical processing. By semi-structured, I mean data that is organized in a tabular format, even though it may not be directly downloadable or available via a web service. There is a preponderance of useful data stored in HTML tables that can be harvested out of a web request. In my case, I wanted to see what effects earnings have on the profitability of entry/exit strategies used for as part of my doctoral research project involving a stock trading simulator. MSN has this great feature called 10 year summary which provides earnings, book value, and other key metrics on an annual basis going back 10 years. For example, you can view Microsoft’s history by going to http://moneycentral.msn.com/investor/invsub/results/compare.asp?Page=TenYearSummary&Symbol=msft.

This gives you the following information for MSFT (Microsoft

	Avg P/E	Price/ Sales	Price/ Book	Net Profit Margin (%)
06/09	13.40	3.66	5.35	24.9
06/08	16.30	4.31	6.94	29.3
06/07	19.90	5.70	8.89	27.5
06/06	21.70	5.54	5.85	28.5
06/05	23.60	6.81	5.53	30.8
06/04	35.70	8.45	4.15	22.2
06/03	36.50	8.67	4.25	23.4
06/02	62.60	10.71	5.62	18.9
06/01	45.90	16.09	8.31	30.5
06/00	53.50	19.29	10.22	41.0

	Book Value/ Share	Debt/ Equity	Return on Equity (%)	Return on Assets (%)	Interest Coverage
06/09	$4.44	0.15	36.8	18.7	NA
06/08	$3.97	0.00	48.7	24.3	NA
06/07	$3.32	0.00	45.2	22.3	NA
06/06	$3.99	0.00	31.4	18.1	NA
06/05	$4.49	0.00	25.5	17.3	NA
06/04	$6.89	0.00	10.9	8.7	NA
06/03	$6.03	0.00	11.6	9.2	NA
06/02	$4.87	0.00	10.3	7.9	NA
06/01	$4.39	0.00	16.3	13.1	NA
06/00	$3.92	0.00	22.8	18.1	NA

Wouldn’t that be handy to have in your financial database? Unfortunately, although MSN provides web services for downloading data, I couldn’t find one that provided this. There are plenty of providers of financial stock information, including Yahoo. Although Yahoo provides an abundance of fundamental data and historical quote information (see http://www.gummy-stuff.org/Yahoo-data.htm for more info), they don’t provide fundamentals data such as Price/Earnings, Book Value on a historical basis.

There are some tools already available to help you interactively copy and paste data from the web directly into spreadsheets such as http://www.inexp.com/table_extractor/default.aspx (DISCLAIMER: None of my recommendations, particularly those about third-party software constitute any endorsement by Microsoft Corporation). However, I couldn’t find any tools to do mass-downloads where you can vary the query string based on parameters from a database.

Here is a look at the raw HTML, for brevity sake, I’ve not included all of it, just that for the first table and the content immediately preceding and subsequent:

<table class="t1"><col class="c1" /><colgroup span="4" class="c2" /><tr class="r0"><th> </th><th>Avg P/E</th><th>Price/ Sales</th><th>Price/ Book</th><th>Net Profit Margin (%)</th></tr><tr><td>06/09</td><td>13.40</td><td>3.66</td><td>5.35</td>
<td>24.9</td></tr><tr><td>06/08</td><td>16.30</td><td>4.31</td><td>6.94</td>
<td>29.3</td></tr><tr><td>06/07</td><td>19.90</td><td>5.70</td><td>8.89</td>
<td>27.5</td></tr><tr><td>06/06</td><td>21.70</td><td>5.54</td><td>5.85</td>
<td>28.5</td></tr><tr><td>06/05</td><td>23.60</td><td>6.81</td><td>5.53</td>
<td>30.8</td></tr><tr><td>06/04</td><td>35.70</td><td>8.45</td><td>4.15</td>
<td>22.2</td></tr><tr><td>06/03</td><td>36.50</td><td>8.67</td><td>4.25</td>
<td>23.4</td></tr><tr><td>06/02</td><td>62.60</td><td>10.71</td><td>5.62</td>
<td>18.9</td></tr><tr><td>06/01</td><td>45.90</td><td>16.09</td><td>8.31</td>
<td>30.5</td></tr><tr><td>06/00</td><td>53.50</td><td>19.29</td><td>10.22</td>
<td>41.0</td></tr></table>

Notice, that the symbol token and name are clearly indicated as part of the URL http://moneycentral.msn.com/investor/invsub/results/compare.asp?Page=TenYearSummary&Symbol=msft. (this is a key enabler for our approach). Furthermore, although using the “View Source” gives us a lot of “gobbly-gook”, note that the data is in HTML tables (another key enabler) and there are definitive identifying tags that surround the data. So, what if we were to write an application that downloaded the content from the web, searched for the identifying tag that starts the table, parsed the table up to point where an ending tag is found and loaded into a database? And what if we wanted to do that for every trading symbol on the AMEX, NYSE, and NASDAQ? Does that sound like an impossible task? Not, if you have a Saturday to spend on it, like I did. That’s right I finished in 1 day, thanks mainly to the power of .NET and SQL, and God’s grace, not particularly due to special skills of my own.

So, here goes, the check list on how to do it:

Define an infrastructure for defining the metadata (search tag to find the table, table columns, etc). about the desired data to download. In my case, I leveraged SQL Server to create some database tables and stored procedures.
Define a component that examines the metadata given a URL and parameter for the query string and returns the metadata as well as receive the metadata, run the web query, and parse the data according to the metadata to return it in a format that can be inserted into a database.
Build the user application that leverages the above infrastructure and components to inserts into the application-specific database.

Step 1 – The Infrastructure for Defining the Metadata

The schema is relatively simple, we just need to be able to identify the URL, the replacement variables in the URL string, and what to look for in each table and how to map them to our database and to our columns. Based on this, here is what I came up with. Everything is varchar except the Ids and sequence fields which are INTs, so no sense in boring you with the full schema:

Based on this table structure, here is the data that is defined to support downloading the MSN history data:

WebUrl:

*ColName*	*ColValue*
UrlId	1
UrlString	http://moneycentral.msn.com/investor/invsub/results/compare.asp?Page=TenYearSummary&Symbol=[Symbol]
SqlSchemaName	load
SqlTableName	Msn10YearSummary
SqlDatabaseName	TradingOptimizer_v3

For the WebUrl table, we simply define the url for the money central 10 year summary along with a placeholder variable for the trading symbol ([Symbol] which will get replaced by the actual symbol when the query is run through the parser component.

UrlVariable

*ColName*	*ColValue*
UrlVariableId	1
UrlId	1
UrlVariableName	[Symbol]
SqlColumnName	Symbol

UrlVariable defines the collection of variables that can be used, which includes all of the variables in the webUrl. This also provides a mapping back to the source SQL Columns. For our scenario, it is overkill, since we only have 1 replacement variable, but if we had multple replacements, such as an effective date range that might vary based on the data, it would be useful.

HtmlTable: For my scenario, I actually have 2 HTML Tables to match to the 2 different tables provided on the MSN site, but let’s just look at the first one.

ColName	ColValue
HtmlTableId	3
UrlId	1
TableStartString	</th><th>Avg P/E</th><th>Price/ Sales</th><th>Price/ Book</th><th>Net Profit Margin (%)</th></tr><tr><td>
TableEndString	</td></tr></table>
TableSequence	1

The Table Start String tells the parsing component what data to look for in order to identify the start of the table. The Table End String indicates when the end of the table has been reached.

HtmlTableColumn:

*HtmlTableColumnId*	*ColumnSequence*	*HtmlTableId*	*ColumnHeader*	*SqlColumnName*
2	1	3	Avg P/E	AvgPe
3	2	3	Price/Sales	PriceSales
4	3	3	Price/Book	PriceBook
5	4	3	Net Profit Magin (%)	NetProfitMarginPct
13	0	3		EffectiveDate

This gives us the sequence for the data in the table along with the mappings to the SQL columns based on the column sequence and header. For example, table item #1, Avg P/e maps to a SQL Column named AvgPe.

Step 2 – The Metadata Examiner/Parser Component

For this, we want to separate the metadata parsing from the actual data parsing. The reason is simply performance, there is no reason to repeatedly parse the database to identify how to find and parse the desired html tables, we can do that on the initial instantiation to construct a schema and then reuse the schema for each instance of the data. When I first prototyped this, I made the mistake of doing it all in one fail swoop, then had to refactor when considering the performance for doing the parsing for all 7,000 symbols I wanted to download.

Based on this, I came up with the following class structure:

Now, you might be wondering about all of the objects for accessing the data; For that I used LINQ-to-SQL and the diagram looks much like the SQL database diagram already shown in step 1. There are a couple of simple stored procedures shown below (generated pretty much as is by my AGE tool (see http://age.codeplex.com/) for doing the retrieval shown below:

/*
==========================================================================================
Author:        Bob L (Generated by Age)
Create date:    8/22/2009
Description:    Selects from UrlVariable
Example:        
==========================================================================================
*/
CREATE PROCEDURE [Web].[SelectUrlVariable]
  @findnull_SqlColumnName bit = 0,
  @find_UrlVariableId int = NULL, 
  @find_UrlId int = NULL, 
  @find_UrlVariableName varchar(500) = NULL, 
  @find_SqlColumnName varchar(128) = NULL
AS
BEGIN
    SET XACT_ABORT ON
    SET NOCOUNT ON

/*
-- ****** DEBUG BEGIN *********
  DECLARE @ParamVar varchar(50)
  SET @ParamVar = 'Test'
-- ****** DEBUG END *********
*/

    -- Want to add additional information to raiserror, so need to save ErrorID
    -- since next statement clears @@ERROR.
    DECLARE @ErrorSave INT
    SET @ErrorSave = 0

  SELECT 
        UrlVariableId,
        UrlId,
        UrlVariableName,
        SqlColumnName
  FROM 
        [UrlVariable]
  WHERE 
        ([UrlVariableId]= @find_UrlVariableId OR @find_UrlVariableId IS NULL )
         AND ([UrlId]= @find_UrlId OR @find_UrlId IS NULL )
         AND ([UrlVariableName]= @find_UrlVariableName 
OR @find_UrlVariableName IS NULL )
         AND ([SqlColumnName]= @find_SqlColumnName 
OR (@find_SqlColumnName IS NULL 
AND @findnull_SqlColumnName= 0) OR ([SqlColumnName] IS NULL 
AND @findnull_SqlColumnName= 1))
    SET @ErrorSave = @@ERROR

    IF @ErrorSave != 0
    BEGIN
      RAISERROR('Failure on SELECT for table [UrlVariable] in 
PROCEDURE [SelectUrlVariable].  Error ID=%d.', 10, 1, @ErrorSave)
      RETURN(@ErrorSave)
    END
END

GO

/*
==========================================================================================
Author:        Bob L (Generated by Age)
Create date:    8/22/2009
Description:    Selects from Url
Example:        
==========================================================================================
*/
CREATE PROCEDURE [Web].[SelectWebUrl]
  @find_UrlId int = NULL, 
  @find_UrlString varchar(500) = NULL, 
  @find_SqlSchemaName varchar(128) = NULL, 
  @find_SqlTableName varchar(128) = NULL, 
  @find_SqlDatabaseName varchar(128) = NULL
AS
BEGIN
    SET XACT_ABORT ON
    SET NOCOUNT ON

/*
-- ****** DEBUG BEGIN *********
  DECLARE @ParamVar varchar(50)
  SET @ParamVar = 'Test'
-- ****** DEBUG END *********
*/

    -- Want to add additional information to raiserror, so need to save ErrorID
    -- since next statement clears @@ERROR.
    DECLARE @ErrorSave INT
    SET @ErrorSave = 0

  SELECT 
        UrlId,
        UrlString,
        SqlSchemaName,
        SqlTableName,
        SqlDatabaseName
  FROM 
        [WebUrl]
  WHERE 
        ([UrlId]= @find_UrlId OR @find_UrlId IS NULL )
         AND ([UrlString]= @find_UrlString OR @find_UrlString IS NULL )
         AND ([SqlSchemaName]= @find_SqlSchemaName 
OR (@find_SqlSchemaName IS NULL))
         AND ([SqlTableName]= @find_SqlTableName 
OR (@find_SqlTableName IS NULL))
         AND ([SqlDatabaseName]= @find_SqlDatabaseName 
OR (@find_SqlDatabaseName IS NULL))
    SET @ErrorSave = @@ERROR

    IF @ErrorSave != 0
    BEGIN
      RAISERROR('Failure on SELECT for table [WebUrl] in PROCEDURE [SelectUrl].  Error ID=%d.', 10, 1, @ErrorSave)
      RETURN(@ErrorSave)
    END
END

GO

So, let’s take a look at the code for this. I use the ETL acronym (Extract-Transform-Load) as the HtmlTable class suffix as this seems a good description for what we are doing with the web data.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Web;
using System.Net;
using System.Data;
using System.Data.Linq;
using Common;

namespace WebEtlHelper
{
    public class HtmlTableEtl
    {
        public string ConnectString { get; set; }
        private WebEtlDataClassesDataContext _dc;
        public HtmlTableEtl(string connectString)
        {
            ConnectString = connectString;
            _dc = new WebEtlDataClassesDataContext(connectString); 
        }

        public List<SelectWebUrlResult> GetUrlInfo (
            string dbName, 
            string schemaName, 
            string tableName)
        {
            List<SelectWebUrlResult> searchResults = new List<SelectWebUrlResult>();
            searchResults = _dc.SelectWebUrl(null, null, schemaName, tableName, dbName).ToList<SelectWebUrlResult>();
            return searchResults;
        }

        public List<SelectUrlVariableResult> GetUrlVariables (int urlId)
        {
            List<SelectUrlVariableResult> searchResults = new List<SelectUrlVariableResult>();
            searchResults = _dc.SelectUrlVariable(false, null, urlId, null, null).ToList<SelectUrlVariableResult>();
            return searchResults;
        }

        public List<HtmlTable> GetHtmlTables (int urlId)
        {
            List<HtmlTable> searchResults = new List<HtmlTable>();
            // Not using stored procedure for this part, because we want to return
            // the complex data type that includes the child entities
            // Maybe need to use "table type" as part of stored procedure return in SQL?
            // Make sure we are sorting by the table sequence and the column sequence

            var options = new System.Data.Linq.DataLoadOptions();
            options.AssociateWith <HtmlTable> (c => c.HtmlTableColumns.OrderBy (p => p.ColumnSequence));
            _dc.LoadOptions = options;
            var htmlTables = from p in _dc.HtmlTables
                             where (p.UrlId == urlId)
                             orderby p.TableSequence
                             select p;
            foreach (HtmlTable htmlTable in htmlTables)
            {
                searchResults.Add(htmlTable);
            }
            return searchResults;
        }

        //public List<SelectHtmlTableColumnResult> GetHtmlTableColumns (int htmlTableId)
        //{
        //    List<SelectHtmlTableColumnResult> searchResults = new List<SelectHtmlTableColumnResult>();
        //    using (WebEtlDataClassesDataContext dc = new WebEtlDataClassesDataContext(ConnectString))
        //    {
        //        searchResults = _dc.SelectHtmlTableColumn(null,htmlTableId).ToList<SelectHtmlTableColumnResult>();
        //    }
        //    return searchResults;
        //}

        public static String ScrapeWebSite(string urlWebQuery)
        {
            WebClient wc = new WebClient();
            string webData = wc.DownloadString(urlWebQuery);
            return webData;
        }

        public static DataTable GetTableData (string webData, HtmlTable htmlTable)
        {
            DataTable dt = new DataTable();
            // At this point, we now have the downloaded web data along with the list of columns 
            // and the information needed to search for the columns
            // First, get rid of special characters including carriage return, tabs, linefeeds
            webData = webData.Replace("\r", "");
            webData = webData.Replace("\n", "");
            webData = webData.Replace("\t", "");
            // Assume we are dealing with non-case sensitive data, so we upper-case everything to make it simpler.
            webData = webData.ToUpper();
        
            int tableStart = webData.IndexOf(htmlTable.TableStartString.ToUpper());
            tableStart = tableStart + htmlTable.TableStartString.Length;
            string tableData = webData.Substring(tableStart);
            int tableEnd = tableData.IndexOf(htmlTable.TableEndString.ToUpper());
            if (tableEnd > 0)
                // We have something to parse
            {
                tableData = tableData.Substring
                    (0,tableEnd);
                // Now comes the hard part
                dt = parseTable(tableData, htmlTable.HtmlTableColumns.ToList<HtmlTableColumn>());
            }
            return dt;
        }

        public static DataTable parseTable(string tableData, List<HtmlTableColumn> htmlTableColumns)
        {
            DataTable dt = new DataTable();
            // To get here, we must have something that shold be parsable and there are no carriage returns
            // We need to split each cell within each row to get the data.
            // First lets split the rows, we do this by converting the table data </TR> to carriage returns
            // The whole string has already been upper-cased at this point.
            tableData = tableData.ToUpper().Replace(@"<TR>", "");
            tableData = tableData.Replace(@"</TD></TR>", "\n");
            tableData = tableData.Replace(@"<TD>", "");
            tableData = tableData.Replace(@"</TD>", "\t");
            string[] rows = tableData.Split('\n');
            // split to colums
            // add the columns to the DataTable
            foreach (HtmlTableColumn htc in htmlTableColumns)
                dt.Columns.Add(htc.SqlColumnName);
            DataRow row = null;
            string[] rowValues;
            object[] rowItems;
            // split the rows
            for (int i = 0; i < rows.Length; i++)
            {
                rowValues = rows[i].Split('\t');
                row = dt.NewRow();
                rowItems = Utility.ConvertStringArrayToObjectArray(rowValues);
                if (rowItems[0] != null && (string)rowItems[0] != "")
                {
                    row.ItemArray = rowItems;
                    dt.Rows.Add(row);
                }
            }
            return dt;
        }



        public static string SetUrlQueryStringValues (string urlString, List<QueryVariable> queryVariables)
        {
            string newUrlString = urlString;
            foreach (QueryVariable qv in queryVariables)
            {
                newUrlString = newUrlString.Replace(qv.VariableName, qv.VariableValue);
            }
            return newUrlString;
        }
    }
}

To go over how to use this, let’s look at the code for our sample application and see how the various methods are leveraged. The following is a summary of steps:

Define the instance-level variables that are shared between our setup logic and the actual parsing.

Instantiate the class
Get the URL Metadata including query parameter replacement variables from Database based on the database/schema/table name we wish to load.
Get the HTML Table definitions and column lists associated with the URL
Setup the URL Variables table to enable the variable replacement in the query string.

First, we create our instance-level variables for storing the metadata

        private List<HtmlTable> _htmlTables;
        // Contains list of all the tables for the HTML along with
        // a child relation for the columns.  This is the metadata
        // that enables parsing the data.

        List<SelectUrlVariableResult> _variableResults;
        // This is the list of query variables applicable
        // to this URL.

        private string _urlTemplateString;
        private void setupDownloadTables()
        {
            // Setup Logging
            TextWriterTraceListener myWriter = new
            TextWriterTraceListener(
                System.IO.File.CreateText("Msn10YearDownloader_"
                + Common.Utility.BuildFileDateTimeStamp() + ".log"));
            Trace.Listeners.Add(myWriter);

Next, instantiate the HtmlTableEtl class

            // Instantiate the Html Table Extract-Transform-Load (ETL) helper HtmlTableEtl
            WebEtlHelper.HtmlTableEtl etl = new HtmlTableEtl
                (Properties.Settings.Default.TradingOptimizer_v3ConnectionString);
            
Get the URL info from the database by passing in the 
desired database, schema, and table name.

            // Get the Url Info 
            List<WebEtlHelper.SelectWebUrlResult> urlResults
                = etl.GetUrlInfo("TradingOptimizer_v3", "load", "Msn10YearSummary");
            // Should only be one

For our scenario, we only have 1 URL, but we could handle multiple URLs

            foreach (SelectWebUrlResult urlResult in urlResults)
            {
                _urlTemplateString = urlResult.UrlString;
                // Setup empty data tables, do this now as a design pattern for batch processing, 
                // rather than embedding in the data loop.  The metadata should only need to be
                // accessed at the beginning of processing.
                // The list of htmlTables should automatically contain the list of columns as
                // well.

Next, we populate the HTML Table definitions based on the URL ID

This will populate a table containing the HTML table structure
as well as the columns that need to be mapped to the database.



                _htmlTables =
                    etl.GetHtmlTables(urlResult.UrlId);
                // Get the variables for the query string (should just be the trading symbol)



_variableResults should now contain the list of query parameters that need to be replaced

                _variableResults = etl.GetUrlVariables(urlResult.UrlId);
                // Now all the plumbing should be in place
            }
        }

Now, that we have the metadata in place, we can use the parsing aspect of the component. Hopefully, the comments in the code are self-explanatory. Basically, the processing consisting of the following steps

Update query variable with passed in value (tradingSymbol)
Download the web data using the query string
Call the ScrapeWebSite method from the HtmlTableEtl class passing in the HtmlTable metadata and the web data.
Iterate through the tables returned from the ScrapeWebSite calls (store them in a list) and set the values to the Linq Entity based on the column mapping. Note, that at this point, we have hard-wired the names to look for, but this could be done generically by generating a dynamic SQL statement.

        private void insertMsn10YearSummary(string tradingSymbol)
        {
            try
            {
                List<DataTable> dts = new List<DataTable>();

                string urlString = _urlTemplateString;
                // Get the variables for the query string (should just be the trading symbol)
                List<QueryVariable> queryVariables = new List<QueryVariable>();
                foreach (SelectUrlVariableResult variable in _variableResults)
                {
                    queryVariables.Add(new QueryVariable(variable.UrlVariableName, tradingSymbol));
                }

                // Update the query string
                urlString = HtmlTableEtl.SetUrlQueryStringValues(urlString, queryVariables);
                // Run the query to get the data, the infrastructure should be in place to 
                // populate the data.
                string webData = HtmlTableEtl.ScrapeWebSite(urlString);
                foreach (HtmlTable htmlTable in _htmlTables)
                {
                    DataTable dt = HtmlTableEtl.GetTableData(
                        webData,
                        htmlTable);
                    dts.Add(dt);
                }

                // Insert the rows into our target table by mapping back to the data table.
                // We are "weak-typed" at this point, not sure best way to make this a 
                // more strongly-typed approach without compromising on the genericity
                // It would be easy enough to use a table adapter and generate a generic SQL
                // insert instead of referencing each column.

                using (MsnDownloadDataClassesDataContext dc = new MsnDownloadDataClassesDataContext())
                {
                    // For this scenario, we already know that there are only 2 tables with the same 
                    // number of rows, so we just iterate the first table and then pull from the second
                    // to build the row to insert
                    int i = 0;
                    List<Msn10YearSummary> msnRows = new List<Msn10YearSummary>();
                    foreach (DataRow dr in dts[0].Rows)
                    {
                        Msn10YearSummary msnRow = new Msn10YearSummary();
                        msnRow.TradingSymbol = tradingSymbol;
                        // Add 2 months to effective date since it can take 60 days for a company to 
                        // announce earnings.
                        msnRow.EffectiveDate =
                            Utility.ConvertMmYytoDate(dr["EffectiveDate"].ToString(), 90).Value.AddMonths(2);
                        msnRow.AvgPe = Utility.ConvertDecimal(dr["AvgPe"]);
                        msnRow.PriceSales = Utility.ConvertDecimal(dr["PriceSales"]);
                        msnRow.PriceBook = Utility.ConvertDecimal(dr["PriceBook"]);
                        msnRow.NetProfitMarginPct = Utility.ConvertDecimal(dr["NetProfitMarginPct"]);

                        // Next set of fields are in the second table.
                        DataRow dr2 = dts[1].Rows[i];
                        msnRow.BookValueShare = Utility.ConvertDecimal(dr2["BookValueShare"]);
                        msnRow.DebtEquity = Utility.ConvertDecimal(dr2["DebtEquity"]);
                        msnRow.ReturnOnEquityPct = Utility.ConvertDecimal(dr2["ReturnOnEquityPct"]);
                        msnRow.ReturnOnAssetsPct = Utility.ConvertDecimal(dr2["ReturnOnAssetsPct"]);
                        msnRow.InterestCoverage = Utility.ConvertDecimal(dr2["InterestCoverage"]);
                        msnRows.Add(msnRow);
                        i++;
                    }
                    dc.Msn10YearSummaries.InsertAllOnSubmit(msnRows);
                    dc.SubmitChanges();
                }
            }
            catch (Exception ex)
            {
                Trace.WriteLine(Utility.buildExceptionMessage("Error downloading info for " + tradingSymbol + "; ", ex));
            }
        }

There are a couple of references to some utility functions that I’ve included below. These help deal with the conversion aspects of the data. Just because we can hit the URL, doesn’t mean that there is actually valid data, in some cases, some of the fields are missing depending on the equity.

        public static decimal? ConvertDecimal(string decimalString)
        {
            decimal? decimalVal = new decimal();
            decimalVal = null;
            if (decimalString != "N/A")
            {
                // Get rid of "$"
                try { decimalVal = Convert.ToDecimal(decimalString.Replace("$","")); }
                catch { }
            }
            return decimalVal;
        }

        public static decimal? ConvertDecimal(object decimalObject)
        {
            return ConvertDecimal(decimalObject.ToString());
        }

        public static Int64? ConvertInt64(string int64String)
        {
            Int64? int64Val = new Int64();
            int64Val = null;
            if (int64String != "N/A")
            {
                // eliminate the decimal point
                // if it ends with "B", then multiply by 1,000,000,000
                // if it ends with M then multiply by 1,000,000
                // if it ends with K then multiply by 1,000
                // Detect the unit of measurement
                decimal factor = 1;
                if (int64String.EndsWith("B"))
                {
                    factor = 1000000000;
                }
                else if (int64String.EndsWith("M"))
                {
                    factor = 1000000;
                }
                else if (int64String.EndsWith("K"))
                {
                    factor = 1000;
                }
                string multiplierString = int64String.Replace("B", "000000000");
                multiplierString = multiplierString.Replace("M", "000000");
                multiplierString = multiplierString.Replace("K", "000");

                try
                {
                    int64Val = (Convert.ToInt64(Convert.ToDecimal
(multiplierString) * Convert.ToDecimal(factor)));
                }
                catch { }
            }
            return int64Val;
        }

        public static DateTime? ConvertMmYytoDate(string mmyy, int maxMillenniumYear)
        {
            // Returns a date extrapolated from a mm/yy string. Sets day to first of month
            // maxMillenniumYear is the furtherest year in the future to assume as being
            // in the 20th millennium.  I.e., if 20 passed and yymm is 03/21 then return 
            // date of 03/01/1921.  If 20 passed and yymm is 03/20 then return 03/01/2020.
            DateTime? convertDate = new DateTime();
            // Assumes 2 digit years and leading 0s are included in the month
            // null is returned if date couldn't be parsed.
            // if parse failure occurs, an exception is raised and must be 
            // handled by calling routine
            mmyy = mmyy.Replace(@"/", "");
            if (mmyy.Length == 4)
            {
                int yyyy = Convert.ToInt16(mmyy.Substring(2, 2));
                yyyy = (yyyy > maxMillenniumYear) ? yyyy + 1900 : 2000 + yyyy;
                int mm = Convert.ToInt16(mmyy.Substring(0, 2));
                convertDate = Convert.ToDateTime(mm.ToString() + "/" + "01" + 
"/" + yyyy.ToString());
            }
            return convertDate;
        }

This is all called through a simple Windows Form App with a button that runs a function to iterate through our LINQ source data source containing all of the equities we want to process.

The download all button runs the below code to invoke the

        private void buttonDownloadAll_Click(object sender, EventArgs e)
        {
            setupDownloadTables();
            using (MsnDownloadDataClassesDataContext dc = 
new MsnDownloadDataClassesDataContext())
            {
                var symbols = from p in dc.EquitySymbols
                              where (p.Tradeable.Value == true)
                              select p;
                foreach (EquitySymbol es in symbols)
                {
                    insertMsn10YearSummary(es.TradingSymbol);
                }
            }
        }

To show you that this really works, here’s the first 50 rows from our downloaded table for the columns defined in the first HTML Table. As I mentioned earlier, not all of the data is available for all of of the equities, hence the nulls in some of the columns. Note that I push the effective date out 2 months from what is reported on the web because it can take that long for companies to report earnings and my simulation correlation analysis has to operate in the context of the reality of what was known at the time of the simulation. Also, the EndDate is a computed column based on adding 1 year and subtracting 1 day, that gives us our nice range intervals ((dateadd(day,(-1),dateadd(year,(1),[EffectiveDate])))

TradingSymbol	EffectiveDate	EndDate	AvgPe	PriceSales	PriceBook
A	12/1/1999	11/30/2000	NULL	NULL	NULL
A	12/1/2000	11/30/2001	44.9	2.12	3.77
A	12/1/2001	11/30/2002	NULL	1.15	1.71
A	12/1/2002	11/30/2003	NULL	NULL	1.31
A	12/1/2003	11/30/2004	NULL	2.49	3.96
A	12/1/2004	11/30/2005	216	2.54	3.23
A	12/1/2005	11/30/2006	62.2	3.22	3.72
A	12/1/2006	11/30/2007	9.8	2.98	3.76
A	12/1/2007	11/30/2008	22.5	2.76	4.22
A	12/1/2008	11/30/2009	17.7	1.43	3.03
AA	2/1/2000	1/31/2001	20.6	1.9	4.83
AA	2/1/2001	1/31/2002	17.7	1.22	2.54
AA	2/1/2002	1/31/2003	35.5	1.36	2.84
AA	2/1/2003	1/31/2004	49.2	0.97	1.94
AA	2/1/2004	1/31/2005	21.7	1.56	2.73
AA	2/1/2005	1/31/2006	21.2	1.22	2.06
AA	2/1/2006	1/31/2007	19.6	1.01	1.92
AA	2/1/2007	1/31/2008	11.9	0.91	1.78
AA	2/1/2008	1/31/2009	11.3	1.08	1.89
AA	2/1/2009	1/31/2010	103.2	0.34	0.77
AA.P	0001-03-01	0002-02-28	NULL	NULL	NULL
AAC	2/1/2000	1/31/2001	NULL	71.73	36.85
AAC	2/1/2001	1/31/2002	NULL	6.66	19.3
AAC	2/1/2002	1/31/2003	NULL	7.37	NULL
AAC	2/1/2003	1/31/2004	NULL	2.99	3.58
AAC	2/1/2004	1/31/2005	1997.5	41.94	6.42
AAC	2/1/2005	1/31/2006	68.9	10.45	4.42
AAC	2/1/2006	1/31/2007	67.7	4.36	1.51
AAC	2/1/2007	1/31/2008	115.7	2	0.97
AAC	2/1/2008	1/31/2009	NULL	1.69	0.58
AAC	2/1/2009	1/31/2010	NULL	0.51	0.17
AACC	2/1/2000	1/31/2001	NULL	NULL	NULL
AACC	2/1/2001	1/31/2002	NULL	NULL	NULL
AACC	2/1/2002	1/31/2003	NULL	NULL	NULL
AACC	2/1/2003	1/31/2004	NULL	NULL	NULL
AACC	2/1/2004	1/31/2005	NULL	NULL	NULL
AACC	2/1/2005	1/31/2006	893.8	3.61	4.02
AACC	2/1/2006	1/31/2007	17	3.31	3.35
AACC	2/1/2007	1/31/2008	14.7	2.42	2.28
AACC	2/1/2008	1/31/2009	22.5	1.37	2.6
AACC	2/1/2009	1/31/2010	19.5	0.67	1.14
AAI	2/1/2000	1/31/2001	NULL	0.56	NULL
AAI	2/1/2001	1/31/2002	6.7	0.8	60.71
AAI	2/1/2002	1/31/2003	NULL	0.67	13.74
AAI	2/1/2003	1/31/2004	33.7	0.39	5.35
AAI	2/1/2004	1/31/2005	8.7	1.12	3.32
AAI	2/1/2005	1/31/2006	150	0.92	2.79
AAI	2/1/2006	1/31/2007	129.5	NULL	4.03
AAI	2/1/2007	1/31/2008	84.6	0.57	2.82
AAI	2/1/2008	1/31/2009	18.8	0.32	1.47

Technorati Tags: Code Project,.NET,Data Mining,Web Data Mining

Nested Identity Keys in SQL?

Bob Leithiser — Mon, 24 Aug 2009 18:32:46 GMT

This weekend I played with partitioning. I have a table with about 10 million rows spanning the last 6 years of stock history for AMEX, NYSE, Indexes, and NASDAQ. I partitioned each year into a separate filegroup so that should make it much easier to purge archive the old years and try to speed up queries on date ranges that involved multiple tables by storage-aligning the indexes. I aligned a related table used for storing calculations and metrics along with aligning an indexed view. Although a few queries actually ran slower, the most critical ones ran faster with the partitioning, since I typically retrieve data grouped together in a fairly tight range.

It seems that in order to really get storage alignment to work well with partitioned tables, you may want to move away from simple surrogate primary keys to compound primary keys where one segment is the partitioning key and the other segment is the unique key. One of the requirements for a partitioning a table is that the partitioning column be part of the clustered key, so unless you are partitioning on what is already the primary clustered key, you’re looking at either no longer using the primary key as the clustering key or creating a compound key. And until recently, the advice has typically been to use simple surrogate primary clustered keys for best performance, which means you would not typically want to partition on an existing surrogate key when moving to partitioning, since by definition a surrogate key generally has no significance, and thus typically not a relevant partitioning parameter.

Therefore partitioning introduces some new dynamics in the way one designs primary keys.

What would be helpful would be to create an identity column grouped on another column (nested). Say I have a partitioning key of year, then I could have my order Id be reset for each year. The advantage is that the overall index space can be reduced since the numbers are only unique within a range for each of the partitioning keys, allowing a smaller data type to be used for the nested identity key.

For example, if I have over 2^32 billion rows (or about 4.2 billion rows), I would have to use a bigint for a surrogate primary key, even if using the negative range (which is unlikely, so the real limit is more like 2.1 billion). If I create 20 partitions, I can now go to 80 billion rows using a simple surrogate integer plus a tinyint partitioning key, assuming non-negative identity values. This means that using the partitioning scheme, I actually use only 5 bytes per unique index value, rather than 8 bytes, and am able to accommodate up to 255 times more data than if using the simple 4 byte key alone.

The other scenario is where you commonly need to keep items in a static order within a parent, i.e. a bill of materials assembly where the line number represents the sequence in which an assembly is to be constructed. To be able to automatically generate sequence numbers guaranteed to be unique within the context of a parent assembly number would be useful.

This could be done with a trigger of course, but that’s not real efficient, and presents more challenges in bulk-loading scenarios.

What I’m thinking of is some syntax along the lines of

CREATE TABLE Table-name

(Grouping-key datatype,

Grouped-Identity-Key datatype identity [start,increment] group by PartitioningId

)

And then you just partition on the Grouping-key for the primary key on a partitioning scheme as in:

ALTER TABLE Table-name

ADD

CONSTRAINT [Primary-key-Constraint-Name] PRIMARY KEY CLUSTERED
(
[Grouping-Key],
[Grouped-Identity-Key]
)

ON [Partition-Scheme](Grouping-Key)

And storage-aligning the other indexes can be done by just referencing the grouping-key in the same way.

This would cause the GroupdIdentityKey to reset back to the initial value each time the Grouping-key changes. The table could then easily be partitioned on the grouping-key, which can be very compact to just do the partitioning, such as the year from an order date.

Thought on this? Is there a way to accomplish this that I am missing?

Technorati Tags: SQL Server 2008,SQL Server Enhancement Suggestions,SQL Partitioning

A Wrapper for Running SQL Server 2008 Reporting Services Reports Anonymously

Bob Leithiser — Thu, 20 Aug 2009 22:23:00 GMT

With SQL Server 2008, Microsoft re-architected SQL Server Reporting Services (SSRS) to no longer have an IIS dependency. It now relies directly on HTTP.SYS and handles web protocols independently. This is a good thing, in a lot of respects (see http://msdn.microsoft.com/en-us/library/bb630410.aspx). But, there is also a down-side for those of us who are used to utilizing the IIS framework for hosting SQL Server reports. For one thing, allowing anonymous access to reports just by configuring an IIS virtual directory mapping is gone. Also, client certificates are no longer supported, which is a big deal if you’re in an environment where these are used (like I am).

There have already been a few articles written about anonymous access for SSRS 2008, and a couple of techniques to allow Report-manager type access (see http://blogs.msdn.com/jameswu/archive/2008/07/15/anonymous-access-in-sql-rs-2008.aspx, http://blogs.msdn.com/brianhartman/archive/2008/11/21/custom-credentials-in-the-report-viewer.aspx). Rather than rehashing them, my purpose in this post is to provide a generic solution for accomplishing anonymous access to your reports, without compromising your server security, and something you can tailor to your environment. Although my goal with this is to simply provide anonymous access, you can use this same technique for other purposes such as allowing smart-card logon to the hosting web site, since essentially the limitations related to accessing SSRS related to the authentication are wrapped in an ASP.NET application that does live off of IIS.

The basic concept is to take the SSRS report manager and wrap up the core functionality for executing reports inside of an ASP.NET web page that dynamically discovers and presents the reports and report folders in the same manner that the Report Manager does and then allow them to be executed. It does not provide all the functionality of Report Manager, such as the report management aspects, but you could tailor this to add that. Obviously for an anonymous-access scenario, most of those management functions don’t make sense, since they are associated with the identity of the user.

I’ve posted the solution on the web site http://www.bobthesis.com/reportsportal/ if you want to actually see how this in action. The menu user-interface is not pretty, but I’m not a web-designer, you can easily improve the cosmetics for that by modifying the master page/styles. The demo uses the Adventure Works samples, but you’ll find as we go through the code, there is nothing specific to Adventure works, other than an entry in the config file that indicates the name of the Adventure Works reports folder as the root folder.

So, let’s step through the solution. There are 7 main components to the solution:

Reporting Services Web Service: The SQL Server Reporting Web Services (ReportingService2005.asmx – that’s right, 2005, it has not been changed for 2008) provides the interface for enumerating the reports and report folders. It provides a host of other capabilities for report execution, rendering and management, but that is the only capability we need for our solution.
The ReportViewer Control: This is a control that we integrate into our ASP.NET solution. It provides the user experience for executing the reports including prompting for the parameters and providing the various options.
A custom class for authenticating to the Reporting Web Service. This is what allows us to wrap the authentication in the ASP.NET page for the Report Viewer control to use to connect to the Reporting Service web service.
An ASP.NET tree-view control for displaying a menu hierarchy for running the reports corresponding to the reporting folder structure defined in SSRS for the particular reporting application.
A master page that includes the tree view control
A default.aspx page that includes code to execute the Reporting Services web service and enumerate the results.
A config file along with entries to define the authentication parameters.

Below is the overall architecture showing the main components. The dashed curved lines represent data flows, solid lines represent composition, dashed lines represent inheritance, and curved non-dashed line represent process flow.

Now, let’s walk through the development of the components, I’m assuming you’re familiar with Visual Studio.NET so will not delve into the details related to the IDE.

First, we create a ASP.NET application project in Visual Studio.NET and we’ll call it ReportsPortal (You could also use a web site project type if you prefer). At this point, we have our web.config and default.aspx. Next, add a master page to the project. At this point, our project structure should like this:

Next, add a reference to the ReportingService2005.asmx. For now, we will use the local machine to create the web service reference. However, we can modify this later to point to a different server, which is what I do for my demo solution. My reporting web service is on a different server than my web server.

Now, our solution should look as follows:

Now, let’s setup our master page to have place holders for our menu and our report viewer. For my demo, I provide the navigation interface on the left side bar and the report viewer on the right side. I use a table control to divide the placeholders and insert the tree navigation control on the left as shown below:

Below is the source code for the Master Page after setting a few properties to make the tree view not completely vanilla:

<%@ Master Language="C#" AutoEventWireup="true" 
CodeBehind="ReportsPortal.master.cs" 
Inherits="ReportsPortal.ReportsPortal" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" 
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" >
<head runat="server">
    <title></title>
   
    </style>
    <style type="text/css">
        .style1
        {
            font-family: "Comic Sans MS";
            height: 351px;
        }
        .style2
        {
            font-family: "Lucida Console";
            font-size: large;
            text-align: center;
        }
    </style>
</head>
<body bgcolor="#f9f9f9">
    <form id="form1" runat="server">
    <div class="style2" 
        style="background-color: #6699FF; font-family: 'Microsoft Sans Serif'; 
font-size: large; color: #800000;">
        Anonymous SQL Server 2008 Reporting Services Wrapper</div>
    <table class="style1">
        <tr valign="top">
            <td bgcolor="#E1FFF7" >
                <asp:TreeView ID="TreeView1" runat="server" 
                        NodeIndent="10" ShowLines="true"  
ImageSet="BulletedList4">
                        <HoverNodeStyle BackColor="#339966" />
                        <SelectedNodeStyle BackColor="#666699" />
                </asp:TreeView>
            </td>
            <td>
        <asp:ContentPlaceHolder ID="ContentPlaceHolder1" runat="server">
        
        </asp:ContentPlaceHolder>
            </td>
        </tr>
    </table>
    </form>
</body>
</html>

Before, we can start writing the code to use the web service to return our report items and before we can start using the Report Viewer Control, let’s put in the “plumbing” in our web.config, add the report viewer control to the the form, ensure we have the necessary reporting services references and leverage the authentication interface.

Here are the web config entries, we create for both defining the starting location for our report menu as well as the user authentication settings for the proxy reporting user. We need to add the ApplicationSettings section to the config file, which I stick on the bottom. Of course, you should use an account wiht minimal user rights as your proxy reporting user, and you may want to use the .NET DPAPI encryption capabilities to encrypt your web.config file (see http://www.dotnetcurry.com/ShowArticle.aspx?ID=185&AspxAutoDetectCookieSupport=1 for more info on encrypting .net config files), although since these are anonymous reports anyways, it isn’t critical, as long as the proxy reporting user has very limited rights. The report user account should be a normal user account and have permissions in the SQL database to the stored procedures, views, tables, required to run the various reports as well as have browser permissions to the Report Server. If you don’t have an instance name remove the “_myinstance” part of the report service url. Notice we have to repeat the server url in 2 different formats – 1 for the web service reference including the reportservice2005.asmx and the other just containing the root location. The second one is used by the authentication interface and it will not work with the full web service url.

<applicationSettings>
  <ReportsPortal.Properties.Settings>
         <setting name="ReportsPortal_WsReportService_ReportingService2005"
              serializeAs="String">
  <value>http://myServer/ReportServer_myinstance/ReportService2005.asmx</value>
          </setting>
          <setting name="MyReportServerUrl" serializeAs="String">
            <value>http://myServer/ReportServer_myinstance</value>
          </setting>
            <setting name="ReportsRootPath" serializeAs="String">
                <value>/</value>
            </setting>
            <setting name="TopLevelMenuText" serializeAs="String">
                <value>AdventureWorks Reports</value>
            </setting>
            <setting name="MyReportViewerPassword" serializeAs="String">
                <value>myPasswordvalue>
            </setting>
            <setting name="myReportViewerUser" serializeAs="String">
                <value>myreportUser</value>
            </setting>
            <setting name="MyReportViewerDomain" serializeAs="String">
                <value>mydomain</value>
            </setting>
      </ReportsPortal.Properties.Settings>
  </applicationSettings>

However, in addition to the above standard application settings, we need to create a special configuration setting for the ReportServerConnection interface in the appsettings section. (see xxx).

Below is in the part of the config file higher up and before system.web:

<appSettings>
        <add key="ReportViewerServerConnection" 
value="MyReportServerConnection, ReportsPortal"/>
</appSettings>

In this case, “My ReportServerConnection” refers to the name of the class file that you create to implement the IreportServerConnection interface and “ReportsPortal” and “ReportsPortal” refers to the name of the assembly for your web application.

I know this is a bit confusing, so I’ve included all of the code in a zip file that you download when you get to the end of the article…)

Note, you can also use the Settings approach to create these and then use the Properties.Default.Settings to access the values rather than using the Configuration Manager method.

Next, let’s add the ReportViewer control to the default.aspx page. Set the property on the default.aspx page to use the ReportsPortal master page, so we should have 1 placeholder to work with and we drag the Microsoft report viewer control from the Reporting section of the toolbox onto the form. When we drag the control on the form, it will update our project to include a reference to the Microsoft.ReportViewer.WebForms assembly as well as update our web.config with the assembly information.

So, now our default.aspx should look like below, we don’t need to set any properties as this will all happen dynamically in the code, however, you will probably want to increase the width and add zoom mode to make the user experience better.

Below is the source of the default.aspx including some property settings for the report viewer rendering:

<%@ Page Language="C#" AutoEventWireup="true" 
CodeBehind="Default.aspx.cs" Inherits="ReportsPortal._Default" 
MasterPageFile="~/ReportsPortal.Master" %>

<%@ Register assembly="Microsoft.ReportViewer.WebForms, 
Version=9.0.0.0, Culture=neutral, 
PublicKeyToken=b03f5f7f11d50a3a" 
namespace="Microsoft.Reporting.WebForms" 
tagprefix="rsweb" %>

<asp:Content ID="Content1" runat="server" 
    contentplaceholderid="ContentPlaceHolder1">
    <asp:Label ID="ReportLabel" runat="server"></asp:Label>
    <rsweb:ReportViewer ID="ReportViewer1" runat="server" 
    SizeToReportContent="True" Width="800px" 
    ZoomMode="PageWidth" Height="600px">
</rsweb:ReportViewer>
</asp:Content>

Now, let’s take a look at the authentication interface portion. I use the ReportServerConnection2 interface because it supports stateless reporting and seems the most robust, there are others that can be used (see http://blogs.msdn.com/brianhartman/archive/2008/11/21/custom-credentials-in-the-report-viewer.aspx). To get it to compile, you will need to add a reference to the Microsoft.ReportViewer class. Here is the code for the MyReportServerConnection.cs class (you can use different names, but remember everything has to be in sync with the config setting that references back to the class).

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using Microsoft.Reporting.WebForms;
using System.Net;
using System.Security.Principal;
using System.Configuration;

namespace ReportsPortal
{
    [Serializable]
    public class MyReportServerConnection : IReportServerConnection2
    {
        public Uri ReportServerUrl
        {
            get
            {
                string url = Properties.Settings.Default.MyReportServerUrl;
                if (string.IsNullOrEmpty(url))
                    throw new Exception("Missing url from the Web.config file");
                return new Uri(url);
            }
        }
        public int Timeout
        {
            // set timeout to 60 seconds
            get { return 60000; }
        }

        public IEnumerable<Cookie> Cookies
        {
            // No custom cookies
            get { return null; }
        }
        public IEnumerable<string> Headers
        {
            // No custom headers
            get { return null; }
        }

        public MyReportServerConnection()
        {
        }


        public WindowsIdentity ImpersonationUser
        {
            get { return null; }
        }

        public ICredentials NetworkCredentials
        {
            get
            {
                // return null will force the use of impersonation, 
                // otherwise, remove the return null and 
                // implement the other app settings to specify the credential details
                // return null;
                string userName = Properties.Settings.Default.myReportViewerUser;
                if (string.IsNullOrEmpty(userName))
                    throw new Exception("Missing user name from Web.config file");
                string password = Properties.Settings.Default.MyReportViewerPassword;
                if (string.IsNullOrEmpty(password))
                    throw new Exception("Missing password from Web.config file");
                string domain = Properties.Settings.Default.MyReportViewerDomain;
                if (string.IsNullOrEmpty(domain))
                    throw new Exception("Missing domain from Web.config file");
                return new NetworkCredential(userName, password, domain);
            }
        }

        public bool GetFormsCredentials(
         out Cookie authCookie, 
         out string userName, 
         out string password, 
         out string authority)
        {
            authCookie = null;
            userName = null;
            password = null;
            authority = null;
            return false;
        }
    }
}

The project should now look something like below:

Now, that the plumbing is done, we can focus on the 2 other pieces of code. First, lets do the code to enumerate the reporting services folder and return the lists into the treeview control. This is the heart of the application as this basically wraps the Report Manager user interface for navigating reports and folders. We put the enumeration code in the Master page .cs (ReportsPortal.Master.cs in our case). The Page_load just clears out the menu and repopulates it from the Report Server by calling BuildNavigation as long as this is not just a postback event. The not postback helps the performance by only doing this when first executing the application. The tree view control will contain all of the folders along with the items arranged hierarchically. Each report item then maps to a navigation url that contains a link back to the default.aspx passing in the full report item path. We can leverage the same credentials from the config file for invoking the report service as we do for running the reports.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using ReportsPortal.WsReportService;
using System.Security;
using System.Net;


namespace ReportsPortal
{
    public partial class ReportsPortal : System.Web.UI.MasterPage
    {
        protected void Page_Load(object sender, EventArgs e)
        {
            if (!Page.IsPostBack)
            {
                TreeView1.Nodes.Clear();
                TreeView1.Nodes.Add(new TreeNode
                    (Properties.Settings.Default.TopLevelMenuText,
                    Properties.Settings.Default.ReportsRootPath));
                BuildNavigation(TreeView1.Nodes[0]);
            }
        }

        private void BuildNavigation(TreeNode parentNode)
        {

            // Enumerate all of the folders and add the reports
            WsReportService.ReportingService2005 ws =
                new WsReportService.ReportingService2005();
// Need to pass in credentials for reporting service.
            NetworkCredential cred = new NetworkCredential(
                    Properties.Settings.Default.myReportViewerUser,
                    Properties.Settings.Default.MyReportViewerPassword,
                    Properties.Settings.Default.MyReportViewerDomain);
            
            ws.Credentials = cred;
            CatalogItem[] items = ws.ListChildren(parentNode.Value, false);
            foreach (CatalogItem item in items)
            {
                if (item.Name != "Data Sources")
                {
                    switch (item.Type)
                    {
                        case ItemTypeEnum.Report:
                            parentNode.ChildNodes.Add
                                (new TreeNode(
                                    item.Name,
                                    item.Path,
                                    null,
                                    "Default.Aspx?Report="
                                    + item.Path, "_top"));
                            break;
                        case ItemTypeEnum.Folder:
                            TreeNode newParentNode = 
new TreeNode(item.Name, item.Path);
                            parentNode.ChildNodes.Add(newParentNode);
                            BuildNavigation(newParentNode);
                            break;
                        default:
                            break;
                    }
                }
            }
            return;
        }

    }
}

Next, we make sure our default.aspx is set Now, we can bring it all together in the default.aspx.cs by doing a page load to evaluate the query string that gets set by the master page navigator when the user selects the link. It then instantiates the MyReportServerConnection class and maps the credentials back to the Server report that needs to be run.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using Microsoft.Reporting.WebForms;
using System.Net;


namespace ReportsPortal
{
    public partial class _Default : System.Web.UI.Page
    {

        protected void Page_Load(object sender, EventArgs e)
        {

            if ((!IsPostBack) && Request.QueryString.Count > 0)
            {
                string reportPath = Request.QueryString[0];
                this.ReportLabel.Text = Request.QueryString[0];
                this.ReportViewer1.ProcessingMode = ProcessingMode.Remote;
                MyReportServerConnection rsc = new MyReportServerConnection();
                this.ReportViewer1.ServerReport.ReportServerCredentials = rsc;
                this.ReportViewer1.ServerReport.ReportPath = reportPath;
                this.ReportViewer1.ServerReport.ReportServerUrl =
                    new Uri((Properties.Settings.Default.MyReportServerUrl));
                this.ReportViewer1.ServerReport.Refresh();
            }
        }
    }
}

Before testing this out, we need to validate that permissions are correct for our demonstration report user. The account will need to have local logon rights for example. If you’re going across firewall boundaries, you’ll probably need to either setup the reports as not requiring credentials and use an execution account, or use SSL with a certificate so you can use basic authentication on the web service without worrying about credentials in the clear.

You can setup an execution account by using the Report Server configuration manager as shown below. If you do this, then you might not even have to bother with the special interface for the connection, but I haven’t tested that yet.

So, here is the end result, Yes, the treeview menu interface isn’t pretty, but it works, and it automatically enumerates down to the lowest folders.

Try it for yourself at http://www.bobthesis.com/reportsportal/

Technorati Tags: SSRS,SQL Server Reporting Services

Cautions with Indexed Views

Bob Leithiser — Sat, 15 Aug 2009 18:32:36 GMT

If you’ve been following my blog, you may remember an article I did extolling the virtues of indexed views - http://blogs.msdn.com/microsoftbob/archive/2009/05/13/sql-data-warehousing-tip-2-indexed-views.aspx. I stand behind that and have found this feature extremely useful. In fact in my last post the 11 seconds it was taking to perform a few thousand transactions against a 10-million row history table with 3 million transaction went down to just 2 seconds using an indexed view.

However, before you go out and make every view into an indexed view, it is important to understand what an indexed view is and what are the consequences. I was reminded of this very sharply when I did the following “routine” update command on my table containing all of the trading symbols used for my simulation processing:

/*------------------------
update dbo.EquitySymbol 
set Shortable = 1 
from Load.ShortableList l
where l.Symbol = dbo.EquitySymbol.TradingSymbol
------------------------*/

(6334 row(s) affected)

Now, normally this had been taking less than 2 seconds. But, this was before I added the indexed view that happened to include the “Shortable” column. So, how long did it take with the indexed view in place?

Almost 19 minutes? Yes, 18 minutes and 51 seconds? That is actually about twice as long than it takes to create the indexed view along with all of it’s alternate indexes! Pardon the drama, but my point is that Indexed views are dramatic by their nature. They are dramatic in that on one hand you can analyze millions of rows and help enabling performing something as complex as generating 7 years of trading simulations consisting of over 3 million orders for 10 million rows in a space of a couple of hours, while on the other hand causing a small update to take over a thousand times longer than usual…

So, how do we explain this? First, we need to understand what happens when you create an indexed view. The first step is to enable schemabinding on the view. This ensures that SQL can tag a dependency between the table and the view so it can prevent and detect changes that might cause the view to stop functioning, that isn’t where the performance hit comes, (see http://msdn.microsoft.com/en-us/library/ms187956.aspx for an explanation of the process for creating an indexed view including the schemabinding step). The key threshold happens (pun intended) when you create the primary key for the view which is the unique clustered index for the view and required before you can add secondary indexes to the view. What that does it “materializes” the view such that it is no longer just a query into your tables, but it becomes a table itself with real data. And of course, that data must be maintained to match the underlying tables.

So, what was the situation in my case? Well, the “shortable” flag was part of the indexed view that joined the equity symbol table (which has about 9,000 rows) to the Equity History table, which happens to have about 10 million rows. My “little” update of the 6,000 or so Equity Symbol rows cascaded into about 7 million updates necessary to maintain all of the rows in the indexed view. Here you can see the major time-consuming aspects of the query plan with the indexed view in place:

Notice the large time for the sorts, these were apparently needed to prepare for secondary index updates that included the shortable column as an included column. And notice that the update time for the clustered index for the table actually containing the column ([EquitySymbol].[PK_EquitySymbol]) is estimated to take virtually 0 percent of the total query time.

So, what does this mean – avoid indexed views? Not at all! It just means as the title suggest: use caution. So what could I have done differently to avoid this issue. To start with, my update command is flawed. I told SQL to update every row and didn’t even look to see if the rows needed to be updated. SQL Server will happily do what you tell it to, it doesn’t have a built-in mechanism that says - “Hey, you don’t need to update that column, it already contains the value you specified”. It doesn’t do that for a very good reason. For example, what if I had a trigger on a table that checked to see if a particular column was updated. Since SQL trigger processing with the “IF UPDATE(column)” doesn’t check to see if the value is actually changed, just whether it is updated, it is easy for me to force a trigger to execute just by updating the column to its current value. So, what I should have done was:

update dbo.EquitySymbol 
set Shortable = 1 
from Load.ShortableList l
where l.Symbol = dbo.EquitySymbol.TradingSymbol
AND dbo.EquitySymbol.Shortable <> 1

Since there were only a handful of rows that had changed since the last time this update command was run, doing it this way, should have taken only a few seconds, albeit it still would have required a couple thousand updates to cascade the updates to the related rows the view. Another option is that if I really did need to update a few thousand rows, would have been to simply drop the indexed view, perform the update, and recreate the indexed again. There may be some more exotic approaches such as disabling the indexes on the indexed view, but effectively the approach is the same.

So, here are a few suggestions when dealing with indexed views:

Do not include more columns than you really need, particularly if the columns may be subject to change.
Consider the cardinality of the values in the indexed view. If your indexed view ends up repeating a few value thousands of times, consider the source of the data. If that data is often changed, then expect those updates to take many times longer than without the indexed view. Be very careful when joining a table with relatively few rows (i.e. 6,000) that is frequently updated to a much larger table (i.e. 10 million), even if the larger table is infrequently updated.
Consider carefully the use of low-cardinality items in the indexes either as segments of the indexes or as included columns, as this will add to the update time for the view.
Don’t create an indexed view unless there is at least one critical query that can benefit significantly and measure that benefit against the cost of other queries that may need to update the table.
When considering the costs/benefit of an indexed view, don’t forget to factor in that SQL will use the indexed view for queries that do not directly reference the indexed view. Absent a query hint to indicate otherwise, SQL Server will leverage the indexed view in query plans that access any tables related to the indexed view. So, when you create an indexed view intended to help one query, you may find it helping other queries as well.

So, the bottom line is that like just about everything else in SQL Server 2008, indexed views have their place, but use them wisely…

Technorati Tags: Indexed Views,SQL Server 2008,SQL Server,SQL Server Performance

Some Uses for Query Hints

Bob Leithiser — Fri, 14 Aug 2009 08:16:00 GMT

Some of you are all set to flame me just because of my title. I know because a couple of years back, I would have probably been one of the first to say “NO!”, when anybody started talking about query hints. But, like everything else in this world, they have their place. First of all, I encourage you to try everything else before resorting to query hints and to avoid hints that specify index names and such directly. The hints I have found to be most useful are fairly generic and do not impose maintenance burdens on the database structure.

The rest of you may be wondering exactly what I’m talking about, so let me explain. When SQL Server evaluates a query, it takes it apart and looks for how it can optimize the retrieval, what indexes can be used. It looks at all sorts of things including selectivity of the indexes, number of physical and logical I/Os required, the parameter values, the search arguments (SARGs) and makes a best-guess stab. Any query you execute, you can select the “Show Query Plan” menu item (or CTRL-L) and get a graphical picture of just how SQL will execute the query. You can also save it in XML format. And it will actually detect “missing indexes”, indexes that would help performance, which I have found incredibly useful and powerful (although I don’t usually follow the index suggestions verbatim – more about that in another post). A query hint is basically telling the query optimizer that you have inside information that is better than what SQL knows. For more background on query plans, query optimization, etc, I suggest the book SQL Server 2008 Internals” by Karen Delaney (http://search.barnesandnoble.com/booksearch/results.asp?ATH=Kalen+Delaney). The SQL team has also created a special blog focused on query optimization at http://blogs.msdn.com/queryoptteam/. Understanding query plans and every single different hints are long blog posts in and of themselves, so I’m not going to attempt to cover this in this article.

My recent experience with query hints came in 2 different areas. Alas, I actually have stats to backup one of my observations. The first observation involved the use of the FAST N hint. This basically tells SQL to optimize only for the first N matches. This can be real useful when using a TOP N where N is a parameter value and you happen to know that it will usually be a small number of rows. SQL doesn’t have the knowledge of that luxury, so forcing an OPTION N hint will cause the optimizer to do SEEKs instead of SCANs which (and I am oversimplifying) means it goes right to the page with the page with the first N matches rather than trying to scan through an entire range or entire table. This can also be useful for partitioning/ranking queries where again you are looking for just the first few rows and not wanting to rank the entire set.

The other experience involved the use of my simulation application. I recently ran the simulations against about 6 years of stock history (almost 10 million rows) in order to simulate different entry/exit trading strategies. This generated nearly 2 million orders and over 1 million end-of-day portfolio rows. What was interesting was that at the beginning of the simulation before there were any rows in my portfolio Orders table, the buy/sell procedures were running very fast. They didn’t start slowing down until over 1 million rows, and even then they weren’t too bad, considering the volume of data. You can see this from the attached chart. At that time, the indexes were undoubtedly not pretty as some of them were being split a lot due to the various combinations of fields that had to be indexed in order to close out positions, evaluate trading effectiveness, etc.

If my routine was simply to generate a couple of million of rows each night, I could just add a post-processing step to rebuild all of the fragmented indexes, then I would have left well enough alone. (for a great tool to automate this, see http://blog.sqlauthority.com/2008/03/04/sql-server-2005-a-simple-way-to-defragment-all-indexes-in-a-database-that-is-fragmented-above-a-declared-threshold/ – I copied and pasted this into a SQL job step I run nightly and I don’t worry any more about index fragmentation and it runs a lot faster than using the built-in SQL index maintenance tools, because it only does what it needs to). In any case, turns out that what I really want to do is just update the data daily with a few new simulations and append to the existing one. For the append, the query plan was still good, since I still need to create a few thousand orders and update a few thousand (fill the buy orders, close the sell orders). But, for just adding a new simulation that goes back 7 years, it was taking inordinately long given the amount of data. You can see from the below chart, that even though only a few or in some cases no records were updated, the buy and sell procedures were taking almost as long as when 2 – 3,000 rows were being updated.


	Main simulation		1 Portfolio Simulation
MarketDate	ElapsedTime	Rows	ElapsedTime	Rows
8/5/2009	11	2466	11	70
8/6/2009	11	2895	11	36
8/7/2009	11	2681	11	206
8/10/2009	11	1603	11	22
8/11/2009	11	2820	12	91
8/12/2009	13	1925	11	29

So, the question is, how can I still keep the batch-oriented performance optimized to process thousands of buy/sell orders all at once, while also getting good-turn around when we have a handful of orders? Enter the query hint “OPTION RECOMPILE”. Most of the time, using this hint doesn’t make sense, because it defeats one of the great features of SQL Server for auto-parameterizing execution. Auto-parameterization is another complex topic covered in the earlier links I listed, but basically allows SQL to look at similar queries, particularly when they are done in a stored procedure, and come up with a reusable plan. It does this based on the parameters for the execution. This works great if you’re calling a stored procedure repeatedly at short intervals, since it saves the recompilation step. However, in my case, my process is mainly batch, I am only calling my stored procedures once per market date for all of the simulations. With the exception of 1 row-oriented procedure, I basically simulate an entire' days worth of stock market activity in 7 stored procedures, so in my scenario I don’t care about the time being saved on the recompile. By using OPTION RECOMPILE, SQL will look at the stored procedure each time and look at all of the parameters and their values and then make the right decisions.

Lets take a look at the stored procedure. You can see for example in my stored procedure where dynamic evaluation of the parameter values would help. If for example, I am calling for just a single portfolio, that makes for an entirely different plan than when there are hundreds of portfolios. One approach involves scanning through the orders by key to look for qualifying orders and then matching back to the portfolios while the other approach involves looking up a single portfolio and evaluating only the orders for that portfolio.

CREATE PROCEDURE [Olap].[ProcessSellOrders]
    @MarketDate DATE, @RowCount BIGINT OUT
AS
BEGIN
    -- SET NOCOUNT ON added to prevent extra result sets from
    -- interfering with SELECT statements.
    SET NOCOUNT ON;
    SET XACT_ABORT ON;
    -- Assume bollinger history technique for now
    -- Similar logic as Process Buys, except reverse the trading methods with same
    -- logic for daily high/daily low, but update Close date, limit instead of Open date, 
    -- and check stop limits 
    
    -- We don't know if the low or high hit first, so to be on the safe side, we assume
    -- that the worse happened and so check the stop limit first 
    UPDATE [Olap].[PortfolioOrder]
        SET FilledPriceClose = dbo.udf_CalculateFillPrice
            (PortfolioOrder.StoplimitPrice
            ,PortfolioOrder.LimitPriceClose
            ,ed.PriceAtOpen
            ,ed.DailyLow
            ,ed.DailyHigh
            ,PortfolioOrder.TradingMethod),
        CloseDate = @MarketDate
    FROM    olap.Portfolio pf, 
            dbo.view_EquityDetails_Brief ed
    WHERE    (pf.StartDate <= @MarketDate and (pf.EndDate >= @MarketDate or pf.EndDate IS NULL))
        AND (pf.LastUpdateDate < @MarketDate or pf.LastUpdateDate is null)
        -- Last Update Date not set until EOD
        AND PortfolioOrder.PortfolioId = pf.PortfolioId
        AND ed.MarketDate = @MarketDate
        AND ed.TradingSymbol = PortfolioOrder.TradingSymbol    
        AND PortfolioOrder.OpenDate <= @MarketDate 
        -- See if valid for day trade - An equity is valid for day trade if it was bought using the
        -- opening price.
        -- Don't get confused by new orders for next market date
        AND PortfolioOrder.CreateDate < @MarketDate 
        -- Redundant since earlier than OpenDate, but include so that optimal index can be used
        AND PortfolioOrder.OrderStatus = 2
        AND (       (ed.PriceAtOpen >= PortfolioOrder.StopLimitPrice AND PortfolioOrder.TradingMethod = 'S' )
-- Stop price hit at open
                OR (ed.PriceAtOpen <= PortfolioOrder.LimitPriceClose AND PortfolioOrder.TradingMethod = 'S' ) 
-- Limit price hit at open
                OR (ed.DailyHigh >= PortfolioOrder.StopLimitPrice AND PortfolioOrder.TradingMethod = 'S' )
-- Stop price hit during the day
                OR (ed.DailyLow <= PortfolioOrder.LimitPriceClose AND PortfolioOrder.TradingMethod = 'S' ) 
-- Limit price hit during the day
            
                OR (ed.PriceAtOpen <= PortfolioOrder.StopLimitPrice AND PortfolioOrder.TradingMethod = 'L')
-- Stop price hit at open
                OR (ed.PriceAtOpen >= PortfolioOrder.LimitPriceClose AND PortfolioOrder.TradingMethod = 'L') 
-- Limit price hit at open
                OR (ed.DailyLow <= PortfolioOrder.StopLimitPrice AND PortfolioOrder.TradingMethod = 'L')
-- Stop price hit during the day
                OR (ed.DailyHigh >= PortfolioOrder.LimitPriceClose AND PortfolioOrder.TradingMethod = 'L')
 -- Limit price hit during the day
            )

            AND (    (PortfolioOrder.IsValidForDayTrade = 1 AND PortfolioOrder.OpenDate = @MarketDate)
                OR (PortfolioOrder.OpenDate < @MarketDate)
            )


    SET @RowCount = @@ROWCOUNT
END

SQL 20008 provides much more flexibility in how to supply query hints. For example, in the above stored procedure, we can just put the keywords OPTION (RECOMPILE) at the end of the query rather than using the legacy RECOMPILE option for the whole stored procedure. If there were multiple queries, we could put different query hints or no query hints with each query, so there is a great improvement in granularity for performing this kind of tweaking in SQL 2008.

Here’s the difference in performance after changing to using the RECOMPILE hint. Times are rounded to the nearest second, so 0 means less than .5 seconds. Here you can see the difference in the query plan when I am only calling for a single portfolio versus the case where all of the portfolios are being processed. OPTION RECOMPILE is particularly useful when there is a lot of variation in the parameters value that ultimately determine the search arguments and predicates for scenarios where one index may make much more sense than another index depending on the search criteria.

	Main simulation		1 Portfolio Simulation		After Option Recompile
MarketDate	ElapsedTime	Rows	ElapsedTime	Rows	ElapsedTime	Rows
8/5/2009	11	2466	11	70	1	70
8/6/2009	11	2895	11	36	0	36
8/7/2009	11	2681	11	206	0	206
8/10/2009	11	1603	11	22	0	22
8/11/2009	11	2820	12	91	0	91
8/12/2009	13	1925	11	29	0	29

P.S: Since I wrote this, I put my indexed view back in and now the sell orders stored procedure even with the main simulation also performs in just a couple of seconds, even with the indexes fragmented.

Technorati Tags: SQL Server 2008,SQL Server Performance,Query Hints

For more about query hints, see http://technet.microsoft.com/en-us/library/ms181714(SQL.90).aspx

Getting the full exception from .NET

Bob Leithiser — Tue, 04 Aug 2009 19:31:46 GMT

While we're on the topic of .NET coding, here's another technique. If you thought my last post was short, here's an even shorter one...

Ever wanted to make sure you returned an error message with all of the inner exceptions? I know I've wasted a lot of time trying to debug code that only returned the top level exception, especially if it occurred in production due to an environmental issue that couldn't be repro-ed in dev, and the production environment didn't allow for Visual Studio debugging.

The below function is a recursive function that calls itself until it has traversed all of the inner exceptions. It will make sure you always get all of the exceptions delimited by a "/" down to the lowest level. The deepest level message will be the last one in the string.

        public static string buildExceptionMessage(String message, Exception ex)
        {
            Exception innerException = ex.InnerException;
            message += ex.Message;
            if (ex.InnerException != null)
            {
                buildExceptionMessage(message + "/", ex.InnerException);
            }
            return message;
        }

Technorati Tags: Visual Studio .NET

Fun with Enums and a Generic File Date/Time Stamper

Bob Leithiser — Tue, 04 Aug 2009 19:26:56 GMT

Time for a break from SQL stuff for a little .NET. Have you ever wanted to use the integer value of an ENUM instead of the actual Enum Value? For example, here is an enum I created to map datetime formats.

public enum FileDatePrecision
{
    Day = 0, Hour = 2, Minute = 4, Second = 6,  
    Secondsf1 = 7, Secondsf2 = 8, Secondsf3 = 9, 
    Secondsf4 = 10, Secondsf5 = 11, Secondsf6 = 12, 
    Secondsf7 = 13
}

Never mind the weird names for the fractional second parts, was going to do "Millisecond, Microsecond, etc, but couldn't figure out the names for all of the fractional precisions. Each "N" value for f is the fractional precision - i.e. f1 means 1/10 of a second, F2 means 1/100, and so on.

So, why did I create the enums with non-contiguous integer values? The reason is to accommodate a simple way to generate the format mask. Here's the code that returns the date time stamp with a couple of overload methods to provide defaults. It's pretty simple, just use the enum value as the substring length in order to include the characters in the format mask needed to achieve the desired output precision.

public static string BuildFileDateTimeStamp(DateTime dt, FileDatePrecision precision)
{
    // Convert the enum to it's integer representation
    int precisonSequence = (int)Enum.Parse(typeof(FileDatePrecision), precision.ToString());
    string formattedTime = "";
    if (precisonSequence > 0)
    {
        // Format the time based on the level of precision specified in the enum
        formattedTime = "_" + dt.ToString("HHmmssfffffff").Substring(0,precisonSequence);
    }
    //to get the date
    string formattedDate = dt.ToString("yyyyMMdd");
    return formattedDate + formattedTime;
}

public static string BuildFileDateTimeStamp(DateTime dt)
{
    return BuildFileDateTimeStamp(dt, FileDatePrecision.Minute);
}

public static string BuildFileDateTimeStamp()
{
    return BuildFileDateTimeStamp(DateTime.Now);
}

So, to create a filename for example for "StockDownloader_20090803.LOG" for use in a trace writer, I can just do

TextWriterTraceListener myWriter = new
      TextWriterTraceListener(
      System.IO.File.CreateText("StockDownloader_" 
      + Common.Utility.BuildFileDateTimeStamp() + ".log"));
    Trace.Listeners.Add(myWriter);

Technorati Tags: Visual Studio .NET

A Generic Interval Range Table to Generate Permutations

Bob Leithiser — Tue, 04 Aug 2009 03:07:18 GMT

In earlier post (http://blogs.msdn.com/microsoftbob/archive/2009/07/06/generating-a-list-of-calendar-dates.aspx), I provided an example of how to use a UDF that returns calendar days in order to generate data from a table valued function. Today, I expand on this using a similar technique, but based on an interval range table which can be used generically. If you've been following my blog, then you're aware of my interest in simulations. The technique that I am going over today is great for this scenario, as we can generate all sorts of combinations of parameters into a result set which can then be joined to our sources to generate the desired combinations of data to test.

To start with, we create a table for storing information about the ranges and how we would like to generate them:

CREATE TABLE [Olap].[IntervalRange](
    [RangeId] [int] IDENTITY(1,1) NOT NULL,
    [RangeStart] [smallmoney] NOT NULL,
    [RangeEnd] [smallmoney] NOT NULL,
    [RangeType] [varchar](5) NOT NULL,
    [RangeInterval] [smallmoney] NOT NULL,
    [RangeGrade] [tinyint] NULL,
 CONSTRAINT [PK_Range] PRIMARY KEY CLUSTERED 
(
    [RangeId] ASC
)) ON [PRIMARY]

Below is some sample data. The RangeStart indicates the first number in the range, the range end indicates the ending value and the range interval determines the increment to use when generating the range.

RangeId	RangeStart	RangeEnd	RangeType	RangeInterval	RangeGrade
3	-6	21	ENTRY	3	NULL
4	25	40	ENTRY	5	NULL
6	2	10	EXIT	2	NULL
7	13	25	EXIT	3	NULL
16	1	5	EPS	1	NULL

From this, we can easily join the data and then insert into a table that contains the combinations to generate a list of strategies which include the EPS Grade, the Exit divisor, and the entry threshold:

    CREATE PROCEDURE [Olap].[GenerateStrategies]
    AS BEGIN
    INSERT INTO Olap.Strategy ( EPSGrade, EntryThreshold, ExitDivisor)
    SELECT eps.Interval as EPSGrade, 
e.Interval as EntryThreshold, 
x.Interval as ExitDivisor 
    FROM Olap.view_RangeIntervals eps 
        CROSS JOIN Olap.view_RangeIntervals e 
        CROSS JOIN Olap.view_RangeIntervals x 
    WHERE eps.RangeType = 'EPS' 
        AND e.RangeType = 'ENTRY' 
        AND x.RangeType = 'EXIT'
    AND NOT EXISTS (SELECT 0 FROM Olap.Strategy
                    WHERE e.Interval = EntryThreshold
                        AND x.Interval = ExitDivisor
                        AND eps.Interval = EPSGrade)
    END

Here are the first few results and the last few results of the cross joins of the parameters. I won't show all of them because it turns out that the select query above actually generates 700 rows in total. This is because there are 5 EPS intervals in total, 14 Entry intervals, and 10 Exit intervals, which works out to 5 X 14 X 10 or 700. So, we basically generate 700 different combinations from our 5 row interval range blueprint data.

RangeType	Interval	RangeType	Interval	RangeType	Interval
EPS	1	ENTRY	-6	EXIT	2
EPS	1	ENTRY	-3	EXIT	2
EPS	1	ENTRY	0	EXIT	2
EPS	1	ENTRY	3	EXIT	2
EPS	1	ENTRY	6	EXIT	2
EPS	1	ENTRY	9	EXIT	2
EPS	1	ENTRY	12	EXIT	2
EPS	1	ENTRY	15	EXIT	2
EPS	1	ENTRY	18	EXIT	2
EPS	1	ENTRY	21	EXIT	2
EPS	1	ENTRY	25	EXIT	2
EPS	1	ENTRY	30	EXIT	2
EPS	1	ENTRY	35	EXIT	2
EPS	1	ENTRY	40	EXIT	2
EPS	1	ENTRY	-6	EXIT	4
EPS	1	ENTRY	-3	EXIT	4
EPS	1	ENTRY	0	EXIT	4
EPS	1	ENTRY	3	EXIT	4
EPS	1	ENTRY	6	EXIT	4
EPS	...	ENTRY	...	...	...

EPS	5	ENTRY	35	EXIT	22
EPS	5	ENTRY	40	EXIT	22
EPS	5	ENTRY	-6	EXIT	25
EPS	5	ENTRY	-3	EXIT	25
EPS	5	ENTRY	0	EXIT	25
EPS	5	ENTRY	3	EXIT	25
EPS	5	ENTRY	6	EXIT	25
EPS	5	ENTRY	9	EXIT	25
EPS	5	ENTRY	12	EXIT	25
EPS	5	ENTRY	15	EXIT	25
EPS	5	ENTRY	18	EXIT	25
EPS	5	ENTRY	21	EXIT	25
EPS	5	ENTRY	25	EXIT	25
EPS	5	ENTRY	30	EXIT	25
EPS	5	ENTRY	35	EXIT	25
EPS	5	ENTRY	40	EXIT	25

There you have it, a whole bunch of permutations from just a small amount of definition data. One caveat is that you may end up with some combinations that are at an extreme and don't make sense for your simulation. To get around this issue, I use an exclusion table that gets generated based on ineffective strategy combinations. This then gets fed back into the process such that strategy permutations are linked via a left outer join, such that those which are found in the exclusion list do not get included in the generation routine.

Technorati Tags: Simulation,SQL UDF,Table Value Function,SQL Server 2008

Using NTILE with Cross Reference Values to Create Dimensions

Bob Leithiser — Tue, 04 Aug 2009 02:42:00 GMT

I confess lack of depth in SQL Analysis Services (SSAS). I've read quite a bit about the capabilities including data mining, and played with a couple of the wizards, but just haven't had time to immerse deeply into end-to-end scenarios. Based on that, I'm writing this post with a big disclaimer - You might be able to do some of what is in this post easier with Analysis Services - I don't really know yet. You'll have to forgive me for ignorance here.

Although SQL Analysis Services provides a lot of capability, it does seem that you will get better results if you "prep" the data before-hand in SQL, particularly the sources for the dimensions. Therefore this post may be useful even if you are highly leveraging SSAS. In my scenario, it would be useful to reduce the range of values associated with the variables, due to the sheer size of the data sets, the multiplicative effect with combining different variable permutations, and for simplicity of the analysis . Basically, this means to summarize ranges of values into different groups and then analyze correlations of the grouped values of the variables - hence the "NTILE" title of this article. Think of it as assigning grades rather than point scores, so you can identify the "A","B" etc students without getting immersed in the point distinctions. Once you have the students grouped by grade levels, then you can much easier do correlative analysis - i.e., how much time do "A" students spend on homework versus ""C" students, as opposed to analyzing the homework correlation with students with 93/100 do compared to 74/100 point-scaled graded students.

First a little bit about the application - My requirements include the ability to correlate various factors about an equity - it's earnings per share (EPS), Price-earnings growth (PEG), Sector, period of performance etc. to indicate if any of these have to do with profitability of a stock purchased at a particular time and when it should be sold.

Hopefully, that provides enough background so some of this will make some sense.

So for my scenario, some what-if questions: Do certain sectors or industries perform better during certain periods in relation to other stocks? How does the EPS affect trading profitability? What about market capitalization? Do the combination of these have an impact? For example, is it more profitable to invest in stocks with small capitalization in some industries during certain times of the year?

Essentially, it would be nice to be able to take all of these factors, generate permutations, and then analyze simulations of stock trades for a period of time to see if any combinations are effective to predict how successful a strategy will perform. Now, if we were to just take all of the combinations of these, the multiplicative effect of the permutations would be unmanageable and more than the "fact" data since each the precision on many of these values is large enough to generate hundreds if not thousands of distinct values. "Permutating" all of this together produces a lot of interesting but meaningless data.

The NTILE feature allows assigning a group based on an interval to a set of data. One of the variables I want to NTILE is Earnings-Per-Share EPS). Another is market capitalization. Another could be Sector. And finally, the period. By NTILE, I mean grouping into categories, so for example, every stock would fall into just 1 of 5 different tiles or ratings for EPS, or for market cap, etc. You can see how by reducing these into groupings, we greatly simplify the analysis and reduce the number of permutations while still having enough granularity to differentiate the groupings. We just have to figure out a way to map the original "fact" data values to their groupings (grades). Once we have the mapping, we can join the tiled dimensional data back to the fact data. Yes, I know this is sounding more like a SSAS scenario, but remember I already gave my disclaimer... I'd welcome responses back from the BI experts on how to approach this scenario using SSAS.

So, here are some steps to accomplishing this:

1) Use computed fields to round the fact data into more discrete values, particularly if the original values are not precise, such as floating point. For example, if we are only need 5 tiles, we probably don't need to map down to the .01 decimal precision. If we instead round down to just the .1 value, our tiling should still be fairly accurate. This reduces the amount of discrete values in the fact data that need to be tiled and reduces the size of the mapping table. Below is an example of an Equity Information table with the computed rounded fields

CREATE TABLE [dbo].[EquityInfo](
    [TradingSymbol] [varchar](25) NOT NULL,
    [Description] [varchar](50) NULL,
    [IndustryId] [smallint] NULL,
    [SectorId] [smallint] NULL,
    [Shares] [bigint] NULL,
    [PERatio] [float] NULL,
    [EPS] [float] NULL,
    [DivYield] [float] NULL,
    [DivPerShare] [float] NULL,
    [PEG] [float] NULL,
    [PriceToSalesRatio] [float] NULL,
    [PriceToBookRatio] [float] NULL,
    [ExchangeName] [varchar](10) NULL,
    [EnabledFlag] [bit] NULL,
    [DateUpdated] [date] NULL,
    [DateCreated] [date] NULL,
    [SampleName] [varchar](50) NULL,
    [TotalBookValue]  AS (case when [PriceToBookRatio]>(0) AND [Shares]>(0) 
then [Shares]/[PriceToBookRatio]  end),
    [TotalSalesValue]  AS (case when [PriceToSalesRatio]>(0) AND [Shares]>(0) 
then [Shares]/[PriceToSalesRatio]  end),
    [PEG_Rounded]  AS (CONVERT([smallmoney],ROUND([PEG],1))) 
PERSISTED,
    [EPS_Rounded]  AS (CONVERT([smallmoney],ROUND([EPS],1))) 
PERSISTED,
    [PE_Rounded]  AS  (CONVERT([smallmoney],ROUND([PERatio],1))) 
PERSISTED,
    [PTB_Rounded]  AS (CONVERT([smallmoney],ROUND([PriceToBookRatio],1))) 
PERSISTED,
    [PTS_Rounded]  AS (CONVERT([smallmoney],ROUND([PriceToSalesRatio],1))) 
PERSISTED,
    [Shortable] [bit] NULL,
    [HistoryReloadRequired] [bit] NULL,
    [HistoryReloadedDateTime] [datetime] NULL,
 CONSTRAINT [PK_EquityInfo] PRIMARY KEY CLUSTERED 
(
    [TradingSymbol] ASC
))

2) Create a view that tiles the values for each of the dimensions of interest. For example, for EPS, we would have the following view:

CREATE VIEW [Olap].[Vdim_EPS_Tiled]
AS
SELECT     NTILE(5) OVER (Order BY EPS_Rounded) as EPS_Tile, EPS_Rounded
FROM         dbo.EquityInfo
WHERE EPS_Rounded Is NOT NULL

3) Generate a table from the view to contain cross-reference the tiles back to the source fact data. The table can be generated through a SELECT INTO and then maintained via a MERGE after we add a primary key. We need to use Select DISTINCT because our first view will contain duplicate EPS_Rounded values.

select distinct * into olap.tdim_EPS_Tiled from Olap.vdim_EPS_Tiled;

GO

ALTER TABLE Olap.tdim_EPS_Tiled ADD CONSTRAINT
    PK_tdim_EPS_Tiled PRIMARY KEY CLUSTERED 
    (EPS_Rounded)

GO

CREATE PROCEDURE olap.Update_tdim_EPS_Tiled
AS BEGIN
  MERGE INTO olap.tdim_EPS_Tiled AS T
  USING olap.vdim_eps_Tiled AS S
    ON S.EPS_Rounded = T.EPS_Rounded
  WHEN MATCHED 
    THEN UPDATE
        SET EPS_Tile = S.EPS_Tile
  WHEN NOT MATCHED BY TARGET
    THEN INSERT
        (EPS_Tile, EPS_Rounded)
        VALUES
        (S.EPS_Tile, S.EPS_Rounded)
  WHEN NOT MATCHED BY SOURCE
    THEN DELETE; -- Might not want to do this if there is history involved
END

Now, whenever I want to analyze performance of an equity related to a simulation, I can easily join to the tdim_EPS_Tiled table on the EPS_Rounded in order to derive the tile. I can then group these, average the profitability from the simulations and evaluate the degree of correlation. This should work well with a cube in SSAS as well to allow correlation and predictive data mining. I'll be trying that out as soon as I get through the SSAS tutorial.

The below diagram illustrates the process flow including the automation aspect.

In summary:

First we reduce the number of values to map, so that instead of mapping the entire universe, we map a rounded representation. We also store it as a precise data type (not floating). This makes our cross-reference tables smaller. This introduces some complexity when we want to map from the fact data back, but this is simply remedied by creating computed fields on the fact data that map back to the source.

Next, we create "Ntile" views of the grouped data. We can now join back from our source data to find the grade that a particular equity is associated with. This allows us to do iterative simulation. By iterative simulation, I mean taking the results of one pass of simulations as input into a higher layer of simulations. For example, if my simulations show a positive correlation for a particular EPS grade, then I can create another simulation that filters just on the EPS grade to drill-down and better explore the correlations from that. good article about iterative and non-iterative simulation algorithms on this link is http://www.stat.columbia.edu/~gelman/research/published/itsim_interface.pdf

You might be wondering why not just use indexed views - I tried that and couldn't get it to work, there are limitations on what you can index when it comes to aggregation and apparently using a function like NTILE is one of them.

Let's all the time I have for this today. You might be thinking this is neat, but what about all my "what-if" questions? How do we query our dimension data and link back to our fact to find the correlations. I have made some interesting correlation discoveries, but I still have a lot of work to do here. I guess this is where I really need to start using SSAS because the queries to actually find the correlations become very complex -- Time to get out the manual and go through the tutorials... Time to start playing with the data mining wizards...

Technorati Tags: SSAS,SQL Analysis Services,Data Warehouse,Data Warehousing,SQL Server 2008,NTILE,MERGE

Using Persisted Computed Columns in SQL Server Indexes

Bob Leithiser — Fri, 10 Jul 2009 23:43:28 GMT

As part of my research work, I generate thousands of simulations on an almost daily basis for various scenarios. One of my scenarios involves determining strategies for selecting the most profitable short and long stock entry/exit positions. Calculating this involves data mining stock history information, analyzing moving averages of the equities, and correlating parameters associated with the equities to find the the optimal entry and exit position for a given stock. I've found one way to calculate this is using brute force computational power to simulate the trades over historic time periods and measure the results. (There probably is an easier way with advanced calculus using derivatives, etc.). In order to play out the simulation, I need to track the position of the equity associated with the order - whether there is an order pending, if the position has been opened, or if the position has been closed, or if the order was cancelled. For example, I need to measure the dollar value of positions as well as cash in the portfolio to tell how much "buying power" remains to purchase additional equities at a given point of time in the simulation.

I track dates associated with all of the events, so a "normalized" way to do this is simply to look at the date as shown in these views:

        1: CREATE VIEW [Simulator].[view_ValidOrders]

       2: AS

       3: SELECT     PortfolioOrderId, PortfolioId, TradingSymbol, CreateDate, OpenDate, UpdateDate, CloseDate, TradingMethod, Shares, LimitPriceOpen, FilledPriceOpen, 

       4:                       LimitPriceClose, FilledPriceClose, StopLimitPrice

       5: FROM         Simulator.PortfolioOrder

       6: WHERE     (CancelDate IS NULL);

       7: CREATE VIEW [Simulator].[view_OpenOrders]

       8: AS

       9: SELECT     PortfolioOrderId, PortfolioId, TradingSymbol, CreateDate, OpenDate, UpdateDate, CloseDate, TradingMethod, Shares, LimitPriceOpen, FilledPriceOpen, 

      10:                       LimitPriceClose, FilledPriceClose, StopLimitPrice

      11: FROM         Simulator.view_ValidOrders

      12: WHERE     ((CloseDate IS NULL) and NOT (OpenDate IS NULL));

      13:  

      14: CREATE VIEW [Simulator].[view_OpenOrders]

      15: AS

      16: SELECT     PortfolioOrderId, PortfolioId, TradingSymbol, CreateDate, OpenDate, UpdateDate, CloseDate, TradingMethod, Shares, LimitPriceOpen, FilledPriceOpen, 

      17:                       LimitPriceClose, FilledPriceClose, StopLimitPrice

      18: FROM         Simulator.view_ValidOrders

      19: WHERE     ((CloseDate IS NULL) and NOT (OpenDate IS NULL));

I can then index on the cancel-date, open-date, and close-date in combination with other fields to try to optimize the queries. But, why do this, when you only need a single status since each state is mutually exclusive? Based on that, we can create a computed field called OrderStatus (we'll come back to the other computed column "Profit" in a moment) that just has the status as in:

CREATE TABLE [Simulator].[PortfolioOrder](
    [PortfolioOrderId] [int] IDENTITY(1,1) NOT NULL,
    [PortfolioId] [int] NOT NULL,
    [TradingSymbol] [varchar](50) NOT NULL,
    [OpenDate] [date] NULL,
    [CloseDate] [date] NULL,
    [CancelDate] [date] NULL,
    [TradingMethod] [char](1) NOT NULL,

    [Shares] [smallmoney] NOT NULL,
    [FilledPriceOpen] [smallmoney] NULL,
    [FilledPriceClose] [smallmoney] NULL,
...
    [Profit]  AS ([shares]*
case 
when [TradingMethod]='L' then [FilledPriceClose]-[FilledPriceOpen] 
else [FilledPriceOpen]-[FilledPriceClose] end) PERSISTED,

    [OrderStatus]  AS (CONVERT([tinyint],
case 
when [CancelDate] IS NOT NULL then (0) 
when [CloseDate] IS NULL AND [OpenDate] IS NULL then (1) 
when [CloseDate] IS NULL AND [OpenDate] IS NOT NULL then (2) 
else (3) end,(0))) PERSISTED,

...

So, how does the computed field help us with performance? First, I need to make sure the computed field is persisted and the caveat on that is that only deterministic values can be computed. Not only that, the computed field must be derived directly from columns in the table, rather than other functions that look at other tables. For example, I have a current value column on my portfolio that is computed based on the value of all of the underlying positions, but that column is not persist-able since it depends on a function that sums the results from the order detail data (I could use an indexed view for this to get around that, but there are quite a few rules and restrictions around indexed views - see my earlier blog on indexed views).

The second thing to do is to make sure that the storage for the computed field is optimal. I wouldn't normally use a floating value for example as a persisted field that gets indexed since floating values are approximate. Also, I want to make sure the computed field uses as little storage as possible, so that my index is as small as possible, thus potentially reducing the number of levels required for a large number of rows. That is why I use the CONVERT function for the order status - by default, SQL would set this to be an integer field which is 4 bytes, but all I really need is a single byte or TINYINT.

Note, that I also use the included columns technique to only include what is needed. I leverage another persisted computed field called profit rather than the all the source fields needed to calculate the profit so that a covering index will work for my queries, thus eliminating the need to do a lookup back to the base table (see the link at end of article for more info on covering indexes). This keeps the included data small, which also contributes to a smaller size index entry.

Using the profit field as an include field along with other relevant fields, and then using the order status gives us the following index, which is just right for the types of queries we need to run in order to perform a stock trading simulation. This allows us to be able to quickly calculate the profit (or loss) and hence current buying power for the portfolio as well as find orders in a given status (such as newly submitted orders with a limit price) that may need to still be opened into a position or open positions that need to be closed to seal a profit.

Index for finding profit of closed orders for portfolio using computed fields:

CREATE NONCLUSTERED INDEX [IX_PortfolioOrder_Status] ON [Simulator].[PortfolioOrder] 
(
    [PortfolioId] ASC, -- 4 bytes
    [OrderStatus] ASC -- 1 byte
)
INCLUDE ( [Profit]) - 4 bytes

Total: 9 bytes

Contrast that with the below index structure that had additional included columns for (Shares, PriceAtClose, PriceAtOpen instead of just Profit) and included the cancel date to filter out invalid orders, or alternatively did not include any of the date fields, but then had to filter after retrieving by PortfolioId.

Index for finding profit of closed orders for portfolio without benefit of computed fields:

CREATE NONCLUSTERED INDEX [IX_PortfolioOrder_Status] ON [Simulator].[PortfolioOrder] 
(
    [PortfolioId] ASC, -- 4 bytes
    [CloseDate] ASC-- 4 bytes, probably not worth having this as part of index and just incorporate

-- into the include list and filter.
)
INCLUDE ( [FilledPriceAtOpen], [FilledPriceAtClose], [TradingMethod], [Shares]) - 13 bytes

Total: 21 bytes

So, how effective is this technique? I"m sorry I don't have hard numbers before-and-after, but I can honestly tell you that my simulations for this application are running about twice as fast now. I believe the main reason is that the number of index levels was reduced, because I was at a threshold due to the number of rows in the Portfolio Orders. Up to a few hundred thousand rows, the performance was similar, however once we cross that number, we require a second index level due to the sheer length of the index when using the . Each additional level in the index requires another I/O to per row lookup.

This brings up another point about indexes and performance. If you experience an overnight-drop in performance on some queries as more data is added, it's probably because one of your indexes just had to add another level to support the binary tree. You'll want to understand how SQL stores indexes to get a better insight on this and other ways to mitigate the issue if merely shrinking the size of the supporting columns is not feasible. Take a look especially at filtered indexing (which we could have used for this scenario, also), where an index can be defined based on horizontal partitioning - i.e. only create this index for rows meeting a specific filter.

This points out the need to carefully consider the lengths of the columns you are indexing as well as the sizes of the fields you include in your include list (at some point it doesn't make sense if you're including every field, may as well forget the list and do a lookup). First, do you really need to use the whole column? If not, you might want to use a computed field to get a substring of what you really need to index. Second, are you using the right data type? If you're shamelessly using GUIDs for primary keys, shame on you, you are wasting 4 times the storage space of a single identity (and not only that creating fragmentation because your inserts are happening all over the tree causing page splits).

I'd like to go deeper into how to calculate number of bytes per row in an index, etc, but don't have time and there are other resources that provide a much better explanation than I can give. For more information, see the following links:

Computed columns including their use in indexes: http://www.mssqltips.com/tip.asp?tip=1702 - Has great explanation and example, better than mine.
Covering indexes: http://msdn.microsoft.com/en-us/library/ms190806.aspx
Indexing tips: http://www.sql-server-performance.com/tips/optimizing_indexes_general_p1.aspx
Why not to use GUIDs if you don't have to: http://www.sqlskills.com/BLOGS/KIMBERLY/post/GUIDs-as-PRIMARY-KEYs-andor-the-clustering-key.aspx
SQL Server Storage Engine: http://blogs.msdn.com/sqlserverstorageengine/archive/2006/06/25/646865.aspx - Includes discussion of storage of included columns.
SQL Server 2008 Internals: http://www.microsoft.com/learning/en/us/book.aspx?ID=12967&locale=en-us (Kalen Delaney, Paul S. Randal, Kimberly L. Tripp, Conor Cunningham, Adam Machanic, and Ben Nevarez)

Technorati Tags: SQL Server,SQL Performance,SQL Server Indexing

Auto-Generating Pivot Views

Bob Leithiser — Fri, 10 Jul 2009 21:22:06 GMT

Coming back from the Sci-Fi world of AI, etc to some real world scenarios... Earlier I posted a generic stored procedure that automatically unpivots data so that columns become rows. Today, I provide the inverse capability, although not truly generic at this point, but the technique is generic.

Recently, I've been working with Microsoft System Center Configuration Management (SCCM) reporting and needed to create a pivot view of a task sequence. If you're not immersed in SCCM, the example probably won't make much sense, but the technique for generating the view may be useful for other scenarios. SCCM provides a lot, (and I mean ALOT) of views and reports out of the box and it provides the ability to create your own reports. SCCM 2007 stores all of it's data in a SQL Server 2005 database along with the views. For more about SCCM, see http://www.microsoft.com/systemcenter/configurationmanager/en/us/default.aspx. For more about creating custom reports, see http://www.microsoft.com/downloads/details.aspx?FamilyId=87BBE64E-5439-4FC8-BECC-DEB372A40F4A&displaylang=en

As much as I'd like to get into SCCM more, let's focus on the pivot view generation which is applicable to any scenario. In our scenario, we have multiple steps for which to report the status for a task sequence. A task sequence is a series of tasks to perform on a selected set of computers (known as a collection). You can think of it as a workflow, and SCCM lets you define whether a sub-task must complete successfully, or may be allowed to fail and allow the task sequence to continue. Task sequences may be re-run on a given machine.

Our business case is that we need to track how well our task sequences are doing and we want to drill down on specific sub-tasks (steps) that have meaning to our specific workflow and provide a single view of how all the sub-tasks performed within a date range.

SCCM captures this information in a view they provide called I created a view on top of this to ensure that only the distinct steps are captured per advertisement)

CREATE VIEW [dbo].[V_Step_Summary] AS
SELECT AdvertisementId, Step, 
SUM(CASE WHEN ExitCode = 0 THEN 1 ELSE 0 END) AS SuccessCount,
SUM(CASE WHEN ExitCode <> 0 THEN 1 ELSE 0 END) AS FailureCount,
COUNT(*) AS AttemptCount
FROM [dbo].[v_TaskExecutionStatus]
GROUP BY AdvertisementId, Step

I also created a view for extracting the status of the steps by date since the underlying data is stored by date/time, and this is SQL 2005, so I can't just down-convert to date. I need to be able to group by date, so that the users can see the count of success/failures in a single day.

CREATE VIEW [dbo].[V_Step_Summary_ByDate] AS
SELECT TOP 100 PERCENT AdvertisementId, Step, 
DATEADD(DD,DATEDIFF(DD,0,ExecutionTime),0) AS ExecutionDate,
SUM(CASE WHEN ExitCode = 0 THEN 1 ELSE 0 END) AS SuccessCount,
SUM(CASE WHEN ExitCode <> 0 THEN 1 ELSE 0 END) AS FailureCount,
COUNT(*) AS AttemptCount
FROM [dbo].[v_TaskExecutionStatus]
GROUP BY AdvertisementId, Step, 
DATEADD(DD,DATEDIFF(DD,0,ExecutionTime),0)

OK, so now we have the building blocks, let's look at the views that pivot the data. We have 3 views: 1 to pivot just the success count, 1 to pivot just the failure count, and 1 to outer join both together to give us both failures and success.

Brace yourself, here are the views:

CREATE VIEW [dbo].[V_StepPivot_SuccessCount] AS   SELECT 
  AdvertisementID, ExecutionDate, 'SuccessCount' as CountType,
 [00] AS [S00],
 [01] AS [S01],
 [02] AS [S02],
 [03] AS [S03],
 [04] AS [S04],
 [05] AS [S05],
 [06] AS [S06],
 [07] AS [S07],
 [08] AS [S08],
 [09] AS [S09],
 [10] AS [S10],
 [11] AS [S11],
 [12] AS [S12],
 [13] AS [S13],
 [14] AS [S14],
 [15] AS [S15],
 [16] AS [S16],
 [17] AS [S17],
 [18] AS [S18],
 [19] AS [S19],
 [20] AS [S20],
 [21] AS [S21],
 [22] AS [S22],
 [23] AS [S23],
 [24] AS [S24],
 [25] AS [S25],
 [26] AS [S26],
 [27] AS [S27],
 [28] AS [S28],
 [29] AS [S29],
 [30] AS [S30],
 [31] AS [S31],
 [32] AS [S32],
 [33] AS [S33],
 [34] AS [S34],
 [35] AS [S35],
 [36] AS [S36],
 [37] AS [S37],
 [38] AS [S38],
 [39] AS [S39],
 [40] AS [S40],
 [41] AS [S41],
 [42] AS [S42],
 [43] AS [S43],
 [44] AS [S44],
 [45] AS [S45],
 [46] AS [S46],
 [47] AS [S47],
 [48] AS [S48],
 [49] AS [S49],
 [50] AS [S50],
 [51] AS [S51],
 [52] AS [S52],
 [53] AS [S53],
 [54] AS [S54],
 [55] AS [S55],
 [56] AS [S56],
 [57] AS [S57],
 [58] AS [S58],
 [59] AS [S59],
 [60] AS [S60],
 [61] AS [S61],
 [62] AS [S62],
 [63] AS [S63],
 [64] AS [S64],
 [65] AS [S65],
 [66] AS [S66],
 [67] AS [S67],
 [68] AS [S68]
  FROM  (SELECT AdvertisementID,   ExecutionDate,Step,SuccessCount FROM dbo.V_Step_Summary_ByDate) p
 PIVOT (SUM (SuccessCount) For Step in (
 [00],
 [01],
 [02],
 [03],
 [04],
 [05],
 [06],
 [07],
 [08],
 [09],
 [10],
 [11],
 [12],
 [13],
 [14],
 [15],
 [16],
 [17],
 [18],
 [19],
 [20],
 [21],
 [22],
 [23],
 [24],
 [25],
 [26],
 [27],
 [28],
 [29],
 [30],
 [31],
 [32],
 [33],
 [34],
 [35],
 [36],
 [37],
 [38],
 [39],
 [40],
 [41],
 [42],
 [43],
 [44],
 [45],
 [46],
 [47],
 [48],
 [49],
 [50],
 [51],
 [52],
 [53],
 [54],
 [55],
 [56],
 [57],
 [58],
 [59],
 [60],
 [61],
 [62],
 [63],
 [64],
 [65],
 [66],
 [67],
 [68]
     )) AS PVT
GO
CREATE VIEW [dbo].[V_StepPivot_FailureCount] AS   SELECT 
  AdvertisementID, ExecutionDate, 'FailureCount' as CountType,
 [00] AS [F00],
 [01] AS [F01],
 [02] AS [F02],
 [03] AS [F03],
 [04] AS [F04],
 [05] AS [F05],
 [06] AS [F06],
 [07] AS [F07],
 [08] AS [F08],
 [09] AS [F09],
 [10] AS [F10],
 [11] AS [F11],
 [12] AS [F12],
 [13] AS [F13],
 [14] AS [F14],
 [15] AS [F15],
 [16] AS [F16],
 [17] AS [F17],
 [18] AS [F18],
 [19] AS [F19],
 [20] AS [F20],
 [21] AS [F21],
 [22] AS [F22],
 [23] AS [F23],
 [24] AS [F24],
 [25] AS [F25],
 [26] AS [F26],
 [27] AS [F27],
 [28] AS [F28],
 [29] AS [F29],
 [30] AS [F30],
 [31] AS [F31],
 [32] AS [F32],
 [33] AS [F33],
 [34] AS [F34],
 [35] AS [F35],
 [36] AS [F36],
 [37] AS [F37],
 [38] AS [F38],
 [39] AS [F39],
 [40] AS [F40],
 [41] AS [F41],
 [42] AS [F42],
 [43] AS [F43],
 [44] AS [F44],
 [45] AS [F45],
 [46] AS [F46],
 [47] AS [F47],
 [48] AS [F48],
 [49] AS [F49],
 [50] AS [F50],
 [51] AS [F51],
 [52] AS [F52],
 [53] AS [F53],
 [54] AS [F54],
 [55] AS [F55],
 [56] AS [F56],
 [57] AS [F57],
 [58] AS [F58],
 [59] AS [F59],
 [60] AS [F60],
 [61] AS [F61],
 [62] AS [F62],
 [63] AS [F63],
 [64] AS [F64],
 [65] AS [F65],
 [66] AS [F66],
 [67] AS [F67],
 [68] AS [F68]
  FROM  (SELECT AdvertisementID,   ExecutionDate,Step,FailureCount FROM dbo.V_Step_Summary_ByDate) p
 PIVOT (SUM (FailureCount) For Step in (
 [00],
 [01],
 [02],
 [03],
 [04],
 [05],
 [06],
 [07],
 [08],
 [09],
 [10],
 [11],
 [12],
 [13],
 [14],
 [15],
 [16],
 [17],
 [18],
 [19],
 [20],
 [21],
 [22],
 [23],
 [24],
 [25],
 [26],
 [27],
 [28],
 [29],
 [30],
 [31],
 [32],
 [33],
 [34],
 [35],
 [36],
 [37],
 [38],
 [39],
 [40],
 [41],
 [42],
 [43],
 [44],
 [45],
 [46],
 [47],
 [48],
 [49],
 [50],
 [51],
 [52],
 [53],
 [54],
 [55],
 [56],
 [57],
 [58],
 [59],
 [60],
 [61],
 [62],
 [63],
 [64],
 [65],
 [66],
 [67],
 [68]
     )) AS PVT

CREATE VIEW [dbo].[V_StepPivot_FailureCount] AS   SELECT 
  AdvertisementID, ExecutionDate, 'FailureCount' as CountType,
 [00] AS [F00],
 [01] AS [F01],
 [02] AS [F02],
 [03] AS [F03],
 [04] AS [F04],
 [05] AS [F05],
 [06] AS [F06],
 [07] AS [F07],
 [08] AS [F08],
 [09] AS [F09],
 [10] AS [F10],
 [11] AS [F11],
 [12] AS [F12],
 [13] AS [F13],
 [14] AS [F14],
 [15] AS [F15],
 [16] AS [F16],
 [17] AS [F17],
 [18] AS [F18],
 [19] AS [F19],
 [20] AS [F20],
 [21] AS [F21],
 [22] AS [F22],
 [23] AS [F23],
 [24] AS [F24],
 [25] AS [F25],
 [26] AS [F26],
 [27] AS [F27],
 [28] AS [F28],
 [29] AS [F29],
 [30] AS [F30],
 [31] AS [F31],
 [32] AS [F32],
 [33] AS [F33],
 [34] AS [F34],
 [35] AS [F35],
 [36] AS [F36],
 [37] AS [F37],
 [38] AS [F38],
 [39] AS [F39],
 [40] AS [F40],
 [41] AS [F41],
 [42] AS [F42],
 [43] AS [F43],
 [44] AS [F44],
 [45] AS [F45],
 [46] AS [F46],
 [47] AS [F47],
 [48] AS [F48],
 [49] AS [F49],
 [50] AS [F50],
 [51] AS [F51],
 [52] AS [F52],
 [53] AS [F53],
 [54] AS [F54],
 [55] AS [F55],
 [56] AS [F56],
 [57] AS [F57],
 [58] AS [F58],
 [59] AS [F59],
 [60] AS [F60],
 [61] AS [F61],
 [62] AS [F62],
 [63] AS [F63],
 [64] AS [F64],
 [65] AS [F65],
 [66] AS [F66],
 [67] AS [F67],
 [68] AS [F68]
  FROM  (SELECT AdvertisementID,   ExecutionDate,Step,FailureCount FROM dbo.V_Step_Summary_ByDate) p
 PIVOT (SUM (FailureCount) For Step in (
 [00],
 [01],
 [02],
 [03],
 [04],
 [05],
 [06],
 [07],
 [08],
 [09],
 [10],
 [11],
 [12],
 [13],
 [14],
 [15],
 [16],
 [17],
 [18],
 [19],
 [20],
 [21],
 [22],
 [23],
 [24],
 [25],
 [26],
 [27],
 [28],
 [29],
 [30],
 [31],
 [32],
 [33],
 [34],
 [35],
 [36],
 [37],
 [38],
 [39],
 [40],
 [41],
 [42],
 [43],
 [44],
 [45],
 [46],
 [47],
 [48],
 [49],
 [50],
 [51],
 [52],
 [53],
 [54],
 [55],
 [56],
 [57],
 [58],
 [59],
 [60],
 [61],
 [62],
 [63],
 [64],
 [65],
 [66],
 [67],
 [68]
     )) AS PVT
GO
CREATE VIEW [dbo].[V_StepPivot_Count] AS SELECT sc.AdvertisementID,   sc.ExecutionDate,  
 [S00], [F00],
 [S01], [F01],
 [S02], [F02],
 [S03], [F03],
 [S04], [F04],
 [S05], [F05],
 [S06], [F06],
 [S07], [F07],
 [S08], [F08],
 [S09], [F09],
 [S10], [F10],
 [S11], [F11],
 [S12], [F12],
 [S13], [F13],
 [S14], [F14],
 [S15], [F15],
 [S16], [F16],
 [S17], [F17],
 [S18], [F18],
 [S19], [F19],
 [S20], [F20],
 [S21], [F21],
 [S22], [F22],
 [S23], [F23],
 [S24], [F24],
 [S25], [F25],
 [S26], [F26],
 [S27], [F27],
 [S28], [F28],
 [S29], [F29],
 [S30], [F30],
 [S31], [F31],
 [S32], [F32],
 [S33], [F33],
 [S34], [F34],
 [S35], [F35],
 [S36], [F36],
 [S37], [F37],
 [S38], [F38],
 [S39], [F39],
 [S40], [F40],
 [S41], [F41],
 [S42], [F42],
 [S43], [F43],
 [S44], [F44],
 [S45], [F45],
 [S46], [F46],
 [S47], [F47],
 [S48], [F48],
 [S49], [F49],
 [S50], [F50],
 [S51], [F51],
 [S52], [F52],
 [S53], [F53],
 [S54], [F54],
 [S55], [F55],
 [S56], [F56],
 [S57], [F57],
 [S58], [F58],
 [S59], [F59],
 [S60], [F60],
 [S61], [F61],
 [S62], [F62],
 [S63], [F63],
 [S64], [F64],
 [S65], [F65],
 [S66], [F66],
 [S67], [F67],
 [S68], [F68]
  FROM V_StepPivot_FailureCount fc
   FULL OUTER JOIN V_StepPivot_SuccessCount sc 
    ON sc.AdvertisementId = fc.AdvertisementId
     AND sc.ExecutionDate = fc.ExecutionDate

Wow, 68 columns are required - 1 for each step! That's a lot of pivoting. If you think I typed in all 68 column names, you're wrong! I generated the code using the below stored procedure. I didn't bother using SP_EXECUTESQL - just output and then manually cut and paste the output into a new query window and then run it from there. It wouldn't be that hard to put this all into a variable and then execute the variable to build the view directly from the stored proc. Note the retrieval from V_Step_Summary in order to generate the list of columns

CREATE PROCEDURE [dbo].[USP_Generate_StepPivot_Views_V3] AS 
BEGIN
    SET NOCOUNT ON
    SELECT 'CREATE VIEW V_StepPivot_SuccessCount AS   SELECT ' AS Line
    UNION ALL SELECT '  AdvertisementID, ExecutionDate, ''SuccessCount'' as CountType,'
    UNION ALL select distinct 
      (' [' + (CASE WHEN STEP < 10 THEN '0' + CONVERT(VARCHAR(1), STEP) ELSE CONVERT(VARCHAR(2),Step) END) + ']') 
    +
      (' AS [S' + (CASE WHEN STEP < 10 THEN '0' + CONVERT(VARCHAR(1), STEP) ELSE CONVERT(VARCHAR(2),Step) END) + 
    (CASE WHEN STEP = (SELECT MAX(STEP) FROM V_Step_Summary) THEN ']' ELSE '],' END))
        FROM dbo.V_Step_Summary

    UNION ALL SELECT '  FROM  (SELECT AdvertisementID,   ExecutionDate,Step,SuccessCount FROM dbo.V_Step_Summary_ByDate) p'
    UNION ALL SELECT ' PIVOT (SUM (SuccessCount) For Step in ('
    
--  This part generates the pivoted column list by selecting the distinct rows that 
-- correspond to the pivot columns to create.  The case statement is used to identify
-- the last step.


UNION ALL select distinct 
      (' [' + (CASE WHEN STEP < 10 THEN '0' + CONVERT(VARCHAR(1), STEP) ELSE CONVERT(VARCHAR(2),Step) END) +
    (CASE WHEN STEP = (SELECT MAX(STEP) FROM V_Step_Summary) THEN ']' ELSE '],' END))
        FROM dbo.V_Step_Summary


    UNION ALL SELECT '     )) AS PVT'
    UNION ALL SELECT 'GO'

    UNION ALL SELECT 'CREATE VIEW V_StepPivot_FailureCount AS   SELECT ' AS Line
    UNION ALL SELECT '  AdvertisementID, ExecutionDate, ''FailureCount'' as CountType,'
    UNION ALL select distinct 
      (' [' + (CASE WHEN STEP < 10 THEN '0' + CONVERT(VARCHAR(1), STEP) ELSE CONVERT(VARCHAR(2),Step) END) +']')
    +
      (' AS [F' + (CASE WHEN STEP < 10 THEN '0' + CONVERT(VARCHAR(1), STEP) ELSE CONVERT(VARCHAR(2),Step) END) + 
    (CASE WHEN STEP = (SELECT MAX(STEP) FROM V_Step_Summary) THEN ']' ELSE '],' END))
        FROM dbo.V_Step_Summary
    UNION ALL SELECT '  FROM  (SELECT AdvertisementID,   ExecutionDate,Step,FailureCount FROM dbo.V_Step_Summary_ByDate) p'
    UNION ALL SELECT ' PIVOT (SUM (FailureCount) For Step in ('
-- This part generates the pivoted column list by selecting the distinct rows that 
-- correspond to the pivot columns to create.  The case statement is used to identify
-- the last step.
    UNION ALL select distinct 
      (' [' + (CASE WHEN STEP < 10 THEN '0' + CONVERT(VARCHAR(1), STEP) ELSE CONVERT(VARCHAR(2),Step) END) +
    (CASE WHEN STEP = (SELECT MAX(STEP) FROM V_Step_Summary) THEN ']' ELSE '],' END))
        FROM dbo.V_Step_Summary
    UNION ALL SELECT '     )) AS PVT'

    UNION ALL SELECT 'GO'
    UNION ALL SELECT 'CREATE VIEW V_StepPivot_Count AS SELECT sc.AdvertisementID,   sc.ExecutionDate, '
    UNION ALL select distinct 
      (' [S' + (CASE WHEN STEP < 10 THEN '0' + CONVERT(VARCHAR(1), STEP) ELSE CONVERT(VARCHAR(2),Step) END) + '],' 
    +
      (' [F' + (CASE WHEN STEP < 10 THEN '0' + CONVERT(VARCHAR(1), STEP) ELSE CONVERT(VARCHAR(2),Step) END) 
    +
    (CASE WHEN STEP = (SELECT MAX(STEP) FROM V_Step_Summary) THEN ']' ELSE '],' END)))
        FROM dbo.V_Step_Summary
    UNION ALL SELECT '  FROM V_StepPivot_FailureCount fc'
    UNION ALL SELECT '   FULL OUTER JOIN V_StepPivot_SuccessCount sc '
    UNION ALL SELECT '    ON sc.AdvertisementId = fc.AdvertisementId'
    UNION ALL SELECT '     AND sc.ExecutionDate = fc.ExecutionDate'
    UNION ALL SELECT 'GO'
END

OK, so you ask "how long did it take you to write that stored procedure"? Didn't it take longer than if you'd just written the views by hand? Yes, probably did, but not by much. And I didn't have to spend much time testing the output view, since it was generated directly from the data that defines the view as being correct. Plus what happens if the number of steps change, do I want to change the pivot view by hand each time? For example, we could actually modify the stored proc to put the code into a variable which it then executes, rather than outputting. Once that is in place, we can create a scheduled task to automatically run this stored procedure on a scheduled basis to ensure that the pivot view is always correct.

Technorati Tags: SQL Server,SCCM,Tips

Software writing software

Bob Leithiser — Wed, 08 Jul 2009 22:42:31 GMT

My academic research relates to software automation – for me that means code writing code. Most code generators today are pretty stupid, though. My hypothesis is basically that software should be able to learn from experience so that it can not only generate tedious code, but also discover algorithms based on results of code execution. So, my goal isn’t just to develop software that generates mundane code components like simple data classes or default stored procedures, but build generators that learn from examining the environment along with trial and error in order to write intelligent systems.

To illustrate this point, I am using Towers of Hanoi as a proof-of-concept of a very simple example for how software can learn an algorithm and then write code that includes the algorithm for a problem domain. The recursive algorithm for solving towers of Hanoi is rather simple – see http://www.cs.cmu.edu/~cburch/survey/recurse/hanoiimpl.html. I found an example in C# for solving the puzzle using the recursive algorithm at http://azerdark.blogspot.com/2008/07/c-tower-of-hanoi.html

Here is a walk-through:

Let Hn = # moves for a stack of n disks.

Here is the optimal strategy:

Move top n−1 disks to spare peg. (Hn−1 moves)

Move bottom disk. (1 move)

Move top n−1 to bottom disk. (Hn−1 moves)

Note that: Hn = 2Hn−1 + 1

The # of moves is described by the below recurrence relation:

Solving Tower of Hanoi RR

Hn = 2 Hn−1 + 1

= 2 (2 Hn−2 + 1) + 1 = 22 Hn−2 + 2 + 1

= 22(2 Hn−3 + 1) + 2 + 1 = 23 Hn−3 + 22 + 2 + 1

…

= 2n−1 H1 + 2n−2 + … + 2 + 1

= 2n−1 + 2n−2 + … + 2 + 1 (since H1 = 1)

= 2n − 1

The code is fairly trivial:

procedure Hanoi(n: integer; source, dest, by: char);
Begin
    if (n=1) then
        writeln('Move the plate from ', source, ' to ', dest)
    else begin
        Hanoi(n-1, source, by, dest);
        Hanoi(1, source, dest, by);
        Hanoi(n-1, by, dest, source);
    end;
End;

But, suppose I didn’t know the algorithm and I want to write code that can discover the algorithm for the game (or potentially any other game) by simply playing the game and looking for patterns. To do such a task requires a framework for capturing the moves and analyzing the states of all the pieces (variables) involved. The framework also needs to be able to know the rules are for performing the moves and how to determine if victory is achieved. In order to do this, I could have targeted the Hanoi puzzle specifically, but I want a framework that will work for potentially any game.

Enter the Gaming Automation Framework (GAF). To build this framework, I utilize SQL Server 2008 as my storage engine as well as leveraging functions and triggers in order to support a schema that not only supports playing the game, but also enforces rules on a game-by-game basis and supports functions for checking the status of a game as well as dictating the best moves.

Here is an overview of the GAF Components:

Game Instantiation SQL Server Database
- Schema for game definitions and instantiation of games
- Database trigger framework for invoking functions enforcing rules that are specific to the game being played and for recording state of the game after each move.
- Stored procedures for creating games, game instances, performing moves, and undoing moves.
- Database function framework for defining what constitutes a valid move and what state arrangement defines success (i.e. winning the game).
Game Player/Solver windows form
- Specification of the game to play along with the number of variable instantiations (i.e. for tower of Hanoi we might chose 3 disks, 4 disks, 5 disks, etc. For checkers we might chose 8 or 12 pieces, etc.)
- Function to play game using depth-first-search (DFS) to probe all moves in order to generate a solution.
- Function to analyze the game moves and determine a pattern and generate code to automatically solve the game (I’m still working on this part).

Let’s step through the major components:

Below are the schemas for game and game variable. Currently this only supports contiguous value check for the variables. I plan to improve this by allowing a discrete ranges to be associated with the variables.

CREATE TABLE [Games].[Game](
    [GameId] [int] IDENTITY(1,1) NOT NULL,
    [GameName] [varchar](50) NOT NULL,
    [MoveRuleFunctionName] [varchar](256) NOT NULL, 
–- function to get list of valid moves

    [GameStatusFunctionName] [varchar](256) NOT NULL, 
–- function to check if game over/won/lost

    [BestMoveRuleFunctionName] [varchar](256) NULL,
 CONSTRAINT [PK_Game_1] PRIMARY KEY CLUSTERED 
(
    [GameId] ASC
))

CREATE TABLE [Games].[GameVariable](

-- The “domain” of pieces”
    [GameId] [int] NOT NULL,
    [VariableId] [int] IDENTITY(1,1) NOT NULL,
    [VariableName] [varchar](50) NOT NULL,
    [MinInstances] [int] NOT NULL, -- Mininum number of “pieces”
    [MaxInstances] [int] NOT NULL, – Max number of pieces.
    [SimulationInstances] [int] NOT NULL, 
-– This is the number of brute force simulations
-- to perform in order to find a solution, not currently using.

    [MinValue] [int] NOT NULL, 
–- min value or (range) (i.e. 1 for peg 1 for Hanoi 
-- that the piece interacts with

    [MaxValue] [int] NOT NULL, 
–- max value (i.e. 3 for Hanoi for peg 3

[StartValue] [int] NOT NULL,
-- staring value of the piece – i.e. peg 1 for Hanoi

CONSTRAINT [PK_GameVariable] PRIMARY KEY CLUSTERED
(
[VariableId] ASC
)

So for the Tower of Hanoi, we define 3 variable rows to represent the disks and use the min and max value to control what peg the disks may be placed on.

Game Table Data:

The game table provides a level of abstraction from the specific game by storing metadata about how to play the game, check the status, and win the game in terms of the names of the functions. This allows multiple games to be setup with different rules. The simulation engine can then evaluate for a particular instance of a game which functions apply.

Games.GameVariable

The game variable relates to the game table and provides information on the variables to be used. I use the terms “variable”/“domain”/”game piece” and “domain”, ”value”, “location” interchangeably not just to confuse you, but because these are different ways to think of these entities depending on the type of scenario we are solving for. However, they are analogous and represent the mapping of a variable-value assignment. This could be a game piece/location, but could represent other types of domain/range pairs.

For our tower of Hanoi example, we only use a single variable to represent the disk and use the instance fields to control how many disks there are. In a a game where there are different types of pieces, separate variables would be created. To represent the valid states of the variable – we utilize start value to represent starting position, final value to represent winning position, and min and max to control the range of values. I plan to improve this so that not only can a range of values, but so that discrete sets of values can be used. For Tower of Hanoi, however this suffices.

Here are the tables used to support instances of the game. Note the use of the columns to store game-state for each move. This is the key component used for the automation aspect in order to discover patterns and translate into a solution algorithm.

CREATE TABLE [Games].[Instance](
    [InstanceId] [int] IDENTITY(1,1) NOT NULL,
    [GameId] [int] NOT NULL,
    [GameDescription] [varchar](255) NULL,
    [VariableInstanceCount] [int] NOT NULL,
    [DomainStates] [varchar](max) NULL, 
-– Each char represents a move and the value in the character 
-- is the variable moved. (i.e. disk number)
    [RangeStates] [varchar](max) NULL, 
-– Each char represents a move and the value in the character 
-- is the value mapped to the moved variable (i.e. peg)
    [GameStatus]  AS ([games].[fn_CheckHanoiGameStatus]([InstanceId],(1))),
 CONSTRAINT [PK_GameInstance] PRIMARY KEY CLUSTERED 
(
    [InstanceId] ASC
))

CREATE TABLE [Games].[InstanceMove](
    [InstanceMoveId] [int] IDENTITY(1,1) NOT NULL,
    [MoveNumber] [int] NULL,
    [InstanceId] [int] NOT NULL,
    [InstanceVariableId] [int] NOT NULL,

-- The “piece” that was moved
    [InstanceVariableValue] [int] NOT NULL,

-– The “location” that piece moved to
 CONSTRAINT [PK_GameMove] PRIMARY KEY CLUSTERED 
(
    [InstanceMoveId] ASC
))

CREATE TABLE [Games].[InstanceVariable](

-- Current value for the piece (“domain" or variable) 
-- and location (“range” or value)
    [InstanceVariableId] [int] IDENTITY(1,1) NOT NULL,
    [VariableId] [int] NOT NULL,
    [InstanceId] [int] NOT NULL,
    [VariableCurrentValue] [int] NOT NULL, 
    [InstanceVariableNumber] [int] NOT NULL,
 CONSTRAINT [PK_MovePosition] PRIMARY KEY CLUSTERED 
(
    [InstanceVariableId] ASC
))

CREATE TABLE [Games].[InstanceVariableValueStates](
-- One per combination of domain and range (piece/location)

    [InstanceId] [int] NOT NULL,
    [InstanceVariableNumber] [int] NOT NULL,
    [InstanceVariableValue] [int] NOT NULL,
    [MoveStates] [varchar](max) NOT NULL, 
–- Bit mask indicating if the variable 
-- was moved on the bit corresponding to the move to the specified value.

 CONSTRAINT [PK_InstanceVariableValueStates] PRIMARY KEY CLUSTERED 
(
    [InstanceId] ASC,
    [InstanceVariableNumber] ASC,
    [InstanceVariableValue] ASC

-- Please don’t get mad I’m using a compound primary key instead of generating

–- a surrogate, think of this as mainly a logical design at this point…
)

Here is the relationship diagram that shows how games and game instances inter-relate. The game instance is just that – an instance of a game, we could have multiple instances per games. Note that this isn’t a perfectly normalized design, some redundancy has been put in to avoid having to write complex queries to link game moves and their variables states back to the instance. We can clean that up later.

A trigger automatically sets the move number and sets the values in the Instance InstanceVariable after each move as well as performing some other tasks that we will get into later. Having the latest instance variable value is handy for processes that involve checking rules, etc, because they are interested in the latest state. They are usually interested in the prior state, also, so I’ll probably enhance this to store the prior value as well.

CREATE TRIGGER [Games].[InstanceMove_ITrig]
   ON  [Games].[InstanceMove]
   AFTER INSERT
AS 
BEGIN

   DECLARE @InstanceId INT
    DECLARE @InstanceVariableNumber INT
    DECLARE @InstanceVariableValue INT
    DECLARE @LastVariableNumber INT = –1 
-- To force variable to be evaluated as null

    DECLARE @OldVariableValue INT
    DECLARE @InstanceMoveId INT
    DECLARE @CustomErrorMessage varchar(255)=''
    
    SET XACT_ABORT ON
    
    BEGIN TRANSACTION
    BEGIN TRY 
        -- Get variable information
        SELECT 
            @InstanceId = INSERTED.InstanceId , 
            @InstanceMoveId = INSERTED.InstanceMoveId,
            @InstanceVariableValue= INSERTED.InstanceVariableValue,
            @InstanceVariableNumber = iv.InstanceVariableNumber 
        FROM INSERTED 
            INNER JOIN  Games.InstanceVariable iv
            ON iv.InstanceVariableId = INSERTED.InstanceVariableId
            
        IF (SELECT COUNT(*) FROM Games.InstanceMove 
WHERE InstanceId = @InstanceID) > 1
        BEGIN
            SELECT TOP 1 @LastVariableNumber = InstanceVariableNumber 
            FROM Games.InstanceMove im
            INNER JOIN Games.InstanceVariable iv
            ON im.InstanceVariableId
                = iv.InstanceVariableId
            WHERE im.InstanceId = @InstanceId
            AND im.InstanceMoveId <> @InstanceMoveId        
            ORDER BY MoveNumber DESC
        END

                
        -- Set move number
        UPDATE [Games].[InstanceMove]
            SET MoveNumber = 
                (SELECT COUNT(*) FROM [Games].[InstanceMove] 
WHERE InstanceId = INSERTED.InstanceId)
        FROM INSERTED
        WHERE INSERTED.InstanceMoveId = InstanceMove.InstanceMoveID

        -- TODO: This part of the code is currently specific to Tower of Hanoi,

need to use the MoveResultStatus in order
        -- to check if this is a bad move and use the mappings to display 
-- the error message.  That will 
        -- allow this trigger to work for any game, not just Tower of Hanoi.
        DECLARE @MoveResult INT -- 1 if valid, 0 if invalid, -1 if 
-- no more moves left
        SET @MoveResult = Games.fn_CheckHanoiMove
            (@InstanceId, @InstanceVariableNumber,@InstanceVariableValue, 
@LastVariableNumber,

@OldVariableValue, 1)
        IF @MoveResult <= –2 
-- Ignore -1 error, since that means we just now ran out of moves,

just need to error out if we have used up all moves
        BEGIN
            SET @CustomErrorMessage = 'Invalid Move: ' + 
            (CASE 
                WHEN @MoveResult = -6 THEN 'Out of Moves'
                WHEN @MoveResult = -2 THEN 'Destination peg ' 
+ CONVERT(VARCHAR(2),@InstanceVariableValue) 
+ ' has a smaller disk on it already than disk ' 
+ CONVERT(VARCHAR(2),@InstanceVariableNumber)
                WHEN @MoveResult = -3 THEN 'Source peg ' 
+ CONVERT(VARCHAR(2),@OldVariableValue) 
+ ' has a smaller disk on top than disk ' 
+ CONVERT(VARCHAR(2),@InstanceVariableNumber)
                WHEN @MoveResult = -4 THEN 'Disk ' 
+ CONVERT(VARCHAR(2),@InstanceVariableNumber) 
+ ' moved to same peg ' 
+ CONVERT(VARCHAR(2),@InstanceVariableValue) + ' as it is already on'
                WHEN @MoveResult = -5 THEN 'Disk ' 
+ CONVERT(VARCHAR(2),@InstanceVariableNumber) 
+ ' moved twice in a row'
                WHEN @MoveResult = -7 THEN 
'Biggest disk Must be moved to center peg at the game point in order to win ' 
            END)
            RAISERROR (@CustomErrorMessage,16,1)
        END

        -- Update Instance Variable
        UPDATE Games.InstanceVariable
        SET VariableCurrentValue = Inserted.InstanceVariableValue
        FROM INSERTED 
        INNER JOIN Games.InstanceVariable IV
        ON IV.InstanceVariableId = INSERTED.InstanceVariableId
        
        -- Update bit masks for moves
        UPDATE Games.InstanceVariableValueStates
        SET MoveStates = COALESCE(MoveStates,'') + '1'
        WHERE InstanceId = @InstanceId
         AND InstanceVariableNumber = @InstanceVariableNumber
         AND InstanceVariableValue= @InstanceVariableValue
         
        UPDATE Games.InstanceVariableValueStates
        SET MoveStates = COALESCE(MoveStates,'') + '0'
        WHERE InstanceId = @InstanceId
         AND NOT (InstanceVariableNumber = @InstanceVariableNumber
         AND InstanceVariableValue= @InstanceVariableValue)
        
        UPDATE Games.Instance
        SET 
            DomainStates = COALESCE(DomainStates + ',','') 
+ CONVERT(VARCHAR(2),@InstanceVariableNumber),
            RangeStates = COALESCE(RangeStates + ',','') 
+ CONVERT(VARCHAR(2),@InstanceVariableValue)
        WHERE InstanceId = @InstanceId
        COMMIT TRANSACTION

There are 4 stored procedures used to play the game

InsertInstance – Creates a new game. Based on the passed-in number of instances which gets set on the Game, Instance variables are generated by the trigger.
InsertMove – Inserts a new move which then results in the execution of the InstanceMove_ITrig trigger above. Returns back the status of the game.
UndoMove – Rolls back a move. This is used when the simulator plays a “loosing” move and allows the simulator to try all different game combinations without having to start all over when hitting a dead-end.
GetInstanceStates – Gets the state data from the Instance table for a particular game instance. The states can then be analyzed for patterns to automate solution discovery.

These are all simple stored procedures with the exception of the UndoMove:

CREATE PROCEDURE [Games].[UndoLastMove]
    -- Add the parameters for the stored procedure here
    @InstanceId INT
AS
BEGIN
    SET NOCOUNT ON 
    SET XACT_ABORT ON

    DECLARE @MoveNumber INT = 0
    SELECT @MoveNumber = COUNT(*) FROM Games.InstanceMove 
WHERE InstanceId= @InstanceId
    
    IF @MoveNumber = 0
    BEGIN
        RETURN -- Nothing to do
    END
    
    BEGIN TRANSACTION
    BEGIN TRY 
        
        -- Identify the variable and value affected by the last move 
-- and the last move id
        DECLARE @InstanceVariableId INT = 0
        DECLARE @InstanceVariableValue INT = 0
        DECLARE @InstanceMoveId INT = 0
        DECLARE @PriorInstanceVariableValue INT = 0
        
        -- Find the current variable and value affected
        SELECT @InstanceVariableId = InstanceVariableId, 
@InstanceVariableValue = InstanceVariableValue,

@InstanceMoveId = InstanceMoveId
        FROM Games.InstanceMove
        WHERE InstanceId = @InstanceId
        AND MoveNumber = @MoveNumber

        -- Delete the move
        DELETE FROM Games.InstanceMove
        WHERE InstanceId = @InstanceId
        AND InstanceMoveId = @InstanceMoveId

        -- Find the prior value for the variable, since the most current 
-- is now deleted
        SELECT TOP 1 @PriorInstanceVariableValue = 
            (CASE WHEN InstanceVariableValue IS NULL THEN gv.StartValue 
ELSE InstanceVariableValue END)
        FROM Games.GameVariable gv
        INNER JOIN Games.InstanceVariable iv
        ON iv.VariableId = gv.VariableId    
        LEFT JOIN Games.InstanceMove im
        ON im.InstanceVariableId = iv.InstanceVariableId
        WHERE iv.InstanceId = @InstanceId
        AND iv.InstanceVariableId = @InstanceVariableId
        ORDER BY MoveNumber DESC
        
        -- Drop the last state from Games.Instance
        UPDATE     Games.Instance
        SET DomainStates = 
                (CASE WHEN @MoveNumber > 1 
                    THEN SUBSTRING(DomainStates,1,DATALENGTH(DomainStates) -

CHARINDEX(',',REVERSE(DomainStates))) 
                    ELSE NULL END),
                RangeStates = 
                (CASE WHEN @MoveNumber > 1 
                    THEN SUBSTRING(RangeStates,1,DATALENGTH(RangeStates)  -

CHARINDEX(',',REVERSE(RangeStates)))
                    ELSE NULL END)
        WHERE InstanceId = @InstanceId
        
        -- Drop the last bit all the Games.InstanceVariableValueStates
        UPDATE Games.InstanceVariableValueStates
        SET MoveStates = 
            (CASE WHEN @MoveNumber > 1 THEN SUBSTRING(MoveStates,1,
@MoveNumber -1)  ELSE '' END)
        WHERE InstanceId = @InstanceId
        
        -- Correct the current value on InstanceVariable
        UPDATE Games.InstanceVariable
        SET VariableCurrentValue = @PriorInstanceVariableValue
        WHERE InstanceVariableId = @InstanceVariableId
         
        COMMIT TRANSACTION
    END TRY
    BEGIN CATCH
        ROLLBACK TRANSACTION
        EXECUTE usp_GetErrorInfo 
    END CATCH
END

CREATE FUNCTION [Games].[fn_CheckHanoiGameStatus]
(
    -- Add the parameters for the function here
    @InstanceId INT,
    @PostMoveCheck bit = 0
)
RETURNS INT
AS
BEGIN
    -- Declare the return variable here
    -- -1 = Game lost, 0 = Undecided, 1 = Game won, -2 = Game not started
    DECLARE @ResultVar INT = 0
    DECLARE @NumberOfDisks INT = 0 
    DECLARE @NumberOfMoves INT = 0
    DECLARE @NumberOfDisksOnLastPeg INT = 0
    DECLARE @MaxMoves INT = 0;
    DECLARE @LastMoveVariableId INT = 0;
    DECLARE @TwoMovesBackVariableId INT = 0;
    DECLARE @MinVariableId INT = 0
    
    SELECT @NumberOfMoves =  COUNT(*) FROM Games.InstanceMove 
WHERE InstanceId = @InstanceId
    -- SELECT @NumberOfDisks = COUNT(*) FROM Games.InstanceVariable 
-- WHERE InstanceId = @InstanceId
    SELECT @NumberOfDisks = VariableInstanceCount, @MaxMoves = 
(POWER(2,@NumberOfDisks) - 1) 
        FROM Games.Instance
         WHERE InstanceId = @InstanceId
    
    
    SELECT @MinVariableId = MIN(InstanceVariableId) 
FROM Games.InstanceVariable
    WHERE InstanceId = @InstanceId

    IF @PostMoveCheck = 1
    BEGIN
        SELECT @TwoMovesBackVariableId = InstanceVariableId
        FROM Games.InstanceMove im 
        WHERE MoveNumber = (@NumberOfMoves - 2)
        AND InstanceId = @InstanceId
        
        SELECT @LastMoveVariableId = InstanceVariableId
        FROM Games.InstanceMove
        WHERE MoveNumber = @NumberOfMoves
        AND InstanceId = @InstanceId
        
    END
        
    DECLARE @IsBigDiskOnLastPegYet BIT = 0
    SELECT @IsBigDiskOnLastPegYet = COUNT(*)
        FROM Games.InstanceVariable
        WHERE InstanceVariableNumber = @NumberOfDisks
        AND VariableCurrentValue = 3
        AND InstanceId = @InstanceId
        
    SELECT @NumberOfDisksOnLastPeg = COUNT(*) FROM Games.InstanceVariable 
    WHERE InstanceId = @InstanceId AND VariableCurrentValue = 3
    -- Check for moves less than 2 raised to number of disks and for all disks 
-- on the correct peg.  The rules function keeps disks from being put on in 
-- the wrong order.
    
    DECLARE @NumberOfMovesLeft INT = @MaxMoves - @NumberOfMoves
    
    
    IF   @NumberOfMoves < POWER(2,@NumberOfDisks) 
    AND (@NumberOfDisksOnLastPeg = @NumberOfDisks)
    BEGIN
        SET @ResultVar = 1

    END
    ELSE BEGIN
        DECLARE @MidPointMoves INT = ((@MaxMoves + 1) /2)
        
        IF @NumberOfMoves > @MidPointMoves 
        BEGIN
            IF NOT EXISTS 
            (SELECT 0 FROM Games.InstanceMove im
                INNER JOIN Games.InstanceVariable iv
                ON im.InstanceVariableId = iv.InstanceVariableId
                AND iv.InstanceVariableNumber = @NumberOfDisks
                AND im.InstanceVariableValue = 3
                AND im.MoveNumber = @MidPointMoves)
            BEGIN
                RETURN -1
            END
        END

        IF  
        (@NumberOfMoves + @NumberOfDisks - @NumberOfDisksOnLastPeg) > @MaxMoves
        -- @NumberOfMoves > POWER(2,@NumberOfDisks - @NumberOfDisksOnLastPeg) - 1
        -- Big diks must reach be on third peg by half of the remaining moves
        OR (@IsBigDiskOnLastPegYet = 0 AND @NumberOfMoves > 
(POWER(2,@NumberOfDisks)/2) -1)
        -- OR (@NumberOfMovesLeft > (POWER(2,@NumberOfDisks - @NumberOfDisksOnLastPeg) 
- POWER(2,@NumberOf))
        -- Every other move must be the smallest peg
         OR (@TwoMovesBackVariableId <> @LastMoveVariableId 
            AND @MinVariableId = @TwoMovesBackVariableId AND @PostMoveCheck =1)
        -- The big disk must move to the center peg at the midpoint of the game.

        BEGIN
            SET @ResultVar = -1    
        END
    END
    RETURN @ResultVar

END

Ultimately, I’d like to eliminate the use of the functions altogether and “schematize” this. That is, the rules and game status should be represent-able within a generic schema. This would allow the user to be able to create any game through a generic interface rather than requiring the development of a function. Based on the fact that the rules and status are completely based on states of entities and their inter-relationships, it should be possible to create a schema to represent this.

Below are the functions used for the game move/game status. Although these functions are specific to the game, they are defined at the game level and the intent is to execute them dynamically in the game engine based on the game selected. Therefore, we are still able to have a generic framework for doing the game even though we have create domain-specific code.

The below code does cheat a little bit – it has a couple of “hints” to prevent non-productive moves and terminate the game early. For example, if the user doesn’t move (or can’t move) the bottom disk on the middle peg in the middle move, we know that the game is lost. (we define losing as not solving the puzzle in the minimum moves). We also don’t let the same move to be reversed, since again that means the optimal moves can’t be achieved. These hints can be picked up through examining the winning pattern, and my long-term goal is for these to be incorporated into the “hinting” aspect of the simulator rather than having to be defined ahead of time.

REATE FUNCTION [Games].[fn_CheckHanoiMove]
(
    -- Add the parameters for the function here
    @InstanceId int,
    @VariableNumber int,
    @VariableValue int,
    @LastVariableNumber int = NULL,
    @OldVariableValue int = NULL,
    @PostCheck bit = 0
)
RETURNS INT
AS
BEGIN
    -- Declare the return variable here
    -- 0 = Move Invalid
    -- -1 = Game over - out of moves
    -- 1 = Move Valid
    DECLARE @ResultVar INT = 1
    DECLARE @NumberOfDisks INT = 0 

    IF @LastVariableNumber IS NULL
    BEGIN
        SELECT TOP 1 @LastVariableNumber = InstanceVariableNumber 
            FROM Games.InstanceMove im
            INNER JOIN Games.InstanceVariable iv
            ON im.InstanceVariableId
            = iv.InstanceVariableId
            WHERE im.InstanceId = @InstanceId
            ORDER BY MoveNumber DESC
    END
    IF @OldVariableValue IS NULL
    BEGIN
        SELECT @OldVariableValue = VariableCurrentValue 
FROM Games.InstanceVariable

WHERE InstanceVariableNumber = @VariableNumber
            AND InstanceId = @InstanceId
    END
    
    -- Represents the peg that we were on before
    SELECT @NumberOfDisks = VariableInstanceCount FROM Games.Instance 
WHERE InstanceId = @InstanceId
    
    -- Validate that there is not already a peg with the disk 
-- which is a lower number disk
    -- (Disks are numbered from 1 to N with 1 as the top disk)
    IF (SELECT COUNT(*) FROM Games.InstanceMove

WHERE InstanceId = @InstanceId) = (POWER(2,@NumberOfDisks) - 1 )
    BEGIN -- No more moves after this one
        SET @ResultVar = -1
    END
    IF (SELECT COUNT(*) FROM Games.InstanceMove

WHERE InstanceId = @InstanceId) > (POWER(2,@NumberOfDisks) - 1 )
    BEGIN -- No more moves
        SET @ResultVar = -6
    END
    ELSE BEGIN 
-- Check that destination peg does not already have a smaller peg on it
        IF ((SELECT COUNT(*) FROM Games.InstanceVariable
                        WHERE VariableCurrentValue = @VariableValue
                          AND InstanceVariableNumber< @VariableNumber
                          AND InstanceId = @InstanceId) > 0)
        BEGIN
            SET @ResultVar = -2 -- Destination peg has a bigger disk on it
        END
        
        ELSE BEGIN 
            -- Check that this disk was the top disk on the move-from peg
            IF ((SELECT COUNT(*) FROM Games.InstanceVariable
                        WHERE InstanceVariableNumber < @VariableNumber
                            AND VariableCurrentValue = @OldVariableValue
                            AND InstanceId = @InstanceId) > 0)
            BEGIN
                SET @ResultVar = -3 
-- Source peg had a smaller disk on it.
            END
            ELSE BEGIN
                IF (@VariableValue = @OldVariableValue)
                -- Move must change the position
                BEGIN
                    SET @ResultVar = -4 -- Disk moved to same peg
                END
                ELSE BEGIN
                    IF (@LastVariableNumber = @VariableNumber AND NOT
 @LastVariableNumber IS NULL)
                    BEGIN
                    -- Don't allow nonsense move 
-- Moved same piece twice in a row 
                        SET @ResultVar = -5
                    END
                    ELSE BEGIN
                        
                        DECLARE @NumberOfMoves INT = 0
                        SELECT @NumberOfMoves =  COUNT(*) 
FROM Games.InstanceMove WHERE InstanceId = @InstanceId
                        IF @PostCheck = 0 -- For pre-check, 
-- increment by one so we are on the right move
                        BEGIN
                            SET @NumberOfMoves = @NumberOfMoves + 1
                        END
    
                        DECLARE @MidPointMoves INT = 
POWER(2,@NumberOfDisks) / 2
        
                        IF @NumberOfMoves = @MidPointMoves 
                            AND (@VariableNumber <> @NumberOfDisks 
OR @VariableVAlue <> 3)
                        BEGIN
                        -- Only valid move here is the big disk to last peg
                            SET @ResultVar = -7
                        END
                    END
                END
            END 
        END
    END

    -- Return the result of the function
    RETURN @ResultVar

END

Regarding these functions, with the exception of the Tower of Hanoi rule check, which we can easily convert to a generic fashion by storing the function name in the game row representing Hanoi and using dynamic SQL, all of the code is generic and not specific to any particular type of game. Let’s take a look at the Windows form used to play the game and play it for a simple 2 disk scenario using brute force. In this scenario, we only “lose” the game once before having to backup. You’ll see as we increase the number of disks on the below graphics, that the loss rate increases very rapidly for games with higher number of disks.

At this point, we now have an interface that allows us to play a game (although it is somewhat tailored to Hanoi and requires a little bit more work to make truly generic) and record the results in state data. Given the above input along with the trigger interaction, here is the data generated including the states of the disks and the pegs as well as overall game states using various variable configuration (2 disks, 3 disks, etc.). Note the patterns. The domain states refer to which variable was moved on a move and the range states indicate which peg was altered for a move. For example for the 2-disk scenario, on Move 1, we moved Disk 1 to Peg 3, On Move 2, we moved Disk 2 to Peg 3 and on Move 3, we moved Disk 1 to Peg 3.

Above are the results of 3 to 5 disk scenarios. Remember, the game is being solved through brute-force at this point. We are just trying to simulate different variations in order to find a pattern or general algorithm for solving the game. All the application understands is how to call a stored procedure to make a rule based on another stored procedure that provides a list of all the legal rules, another procedure to find out the status and determine if a game is still winnable or if it has been won, and finally a stored procedure that can be called to back out a move and try again without having to start all over. Somewhere in the list is the required rule, as there is only 1 “winning” solution to any Hanoi game that uses the fewest moves. Using this approach, along with adding a couple of hints to prevent obvious dead-ends game-plays or detect losses half-way through, we can practically simulate up to 6 disks before the time complexity involved with the recursive depth-searches becomes overwhelming.

Below is the main code in the c# program for playing the game. As you can see, there is nothing specific to Hanoi, this could be used to try to play any game and find the solution by brute force.

        public MoveStatus(fn_GetValidHanoiMoves_View move, bool used)
        {
            Move = move;
            Used = used;
        }

        private List<MoveStatus> getMoveStatuses(int instanceId)
        {
            List<MoveStatus> moveStatuses = new List<MoveStatus>();
            {
                var gameMoves = from p in _dc.fn_GetValidHanoiMoves(instanceId)
                                select p;
                foreach (fn_GetValidHanoiMoves_View gameMove in gameMoves)
                {
                    // Add them to the list
                    moveStatuses.Add(new MoveStatus(gameMove, false));
                }
                //_dc.SubmitChanges();
            }
            return moveStatuses;
        }


        private void move(List<MoveStatus> moveStatuses)
        {
            int? gameStatus = 0;
            Stack<List<MoveStatus>> moveStack = new Stack<List<MoveStatus>>();
            while (gameStatus != 1)
            {
                // Create a new list on the stack unless we are in undo mode
                if (moveStack.Count() >= Math.Pow(2.00, 
(float)this.numericUpDownObjects.Value))
                {
                    // Went too far, roll back the stack
                    _dc.UndoLastMove(_InstanceId);
                    moveStatuses = moveStack.Pop();
                }
                bool madeMove = false;
                // Try a move until one works
                foreach (MoveStatus moveStatus in moveStatuses)
                {
                    if (!moveStatus.Used)
                    {
                        int? lastMoveId = 0;
                        moveStatus.Used = true;
                        _attempts++;
                        _dc.InsertInstanceMove(ref lastMoveId, ref gameStatus, 
_InstanceId,
                            moveStatus.Move.InstanceVariableId,
                            moveStatus.Move.InstanceVariableValue);
                        this.listBox1.Items.Add("Move " + 
moveStatus.Move.InstanceVariableNumber.ToString() +
                            " to " 
+ moveStatus.Move.InstanceVariableValue.ToString()

+ "; stack=" + moveStack.Count().ToString());
                        _dc.SubmitChanges();
                        // if move fails, undo it.
                        if (gameStatus == -1)
                        {
                            _losses++;

                            _dc.UndoLastMove(_InstanceId);
                            this.listBox1.Items.Add("Undo; stack=" 
+ moveStack.Count().ToString());
                            _dc.SubmitChanges();
                        }
                        // If move was ok or game-won then break out
                        else
                        {
                            madeMove = true;
                            break;
                        }
                    }
                }
                // If a move worked, we can move on, else we need to backtrack
                if (gameStatus == 0 && madeMove)
                {
                    moveStack.Push(moveStatuses);
                    moveStatuses = (getMoveStatuses(_InstanceId.Value));
                }
                if (gameStatus == -1 || !madeMove || moveStatuses.Count()== 0)
                {
                    // If game status is loosing, or we weren't able to make 
// a move
                    // or the move status count (generated in line just above)
                    // is 0, then we undo and pop the stack back.
                    _dc.UndoLastMove(_InstanceId);
                    this.listBox1.Items.Add("Undo; stack=" 
+ moveStack.Count().ToString());
                    _dc.SubmitChanges();
                    moveStatuses = (moveStack.Pop()); 
// Go back to the prior move statuses
                }
            }
        }

Let’s take a closer look at domain states and range states to understand their significance:

Disks	DomainStates	RangeStates
2	1,2,1	2,3,3
3	1,2,1,3,1,2,1	3,2,2,3,1,3,3
4	1,2,1,3,1,2,1,4,1,2,1,3,1,2,1	2,3,3,2,1,2,2,3,3,1,1,3,2,3,3
5	1,2,1,3,1,2,1,4,1,2,1,3,1,2,1,5,1,2,1,3,1,2,1,4,1,2,1,3,1,2,1	3,2,2,3,1,3,3,2,2,1,1,2,3,2,2,3,1,3,3,1,2,1,1,3,3,2,2,3,1,3,3

Based on the patterns and their interrelationships, we can determine what a solution would be for N disks through some mathematical introspection. We should be able to do this generically in code. For example, note that where N is the number of disks, N is always moved on the T/2 moves where T is the total number of moves required for the optimal solution. Note also that there is symmetry in the domain states before and after the middle move – i.e. for disk-2 scenario, the same disk is moved on the first and final third move, for 3-disk scenario, we have 1,2,1, both before and after the middle-move and this pattern continues for the 4 and 5 disk scenarios. Thus we can generate a hint for an optimal solution to automatically replay the first half of the game when trying to solve for a greater number of disks. There are additional patterns in the range states and how they relate to the domains. By simply finding all of the patterns, one should be able to define the winning pattern for any number of disks without even knowing the recurrence relation.

Another way to look at this is from the point of view of the variable states, which my infrastructure provides through the code previously shown in the trigger. It just involves appending a 1 or 0 on to the bit mask based on whether or not the variable was affected.

        UPDATE Games.InstanceVariableValueStates
        SET MoveStates = COALESCE(MoveStates,'') + '1'
        WHERE InstanceId = @InstanceId
         AND InstanceVariableNumber = @InstanceVariableNumber
         AND InstanceVariableValue= @InstanceVariableValue
         
        UPDATE Games.InstanceVariableValueStates
        SET MoveStates = COALESCE(MoveStates,'') + '0'
        WHERE InstanceId = @InstanceId
         AND NOT (InstanceVariableNumber = @InstanceVariableNumber
         AND InstanceVariableValue= @InstanceVariableValue)

At the end of the game, we end up with the following. the bit mask just indicates what move the disk is moved to a particular peg. For example in instance 200, the 3 disk game, we find that disk 2 is moved onto disk 3 on the second move.

Instance	Variable	Value	Bit Mask (each bit is a move, and a 1 indicate a pairing of the variable (disk) and the value (peg).
199	1	1	000 - Disk 1 never moved to peg 1 (started there, but never moved there)
199	1	2	100 - Disk 1 moved to disk 2 on move 1
199	1	3	001 - Disk 1 moved to disk 3 on move 3 (last move)
199	2	1	000
199	2	2	000
199	2	3	010– Disk 2 moved onto peg 3 on move 2
200	1	1	0000100
200	1	2	0010000
200	1	3	1000001
200	2	1	0000000
200	2	2	0100000
200	2	3	0000010
200	3	1	0000000
200	3	2	0000000
200	3	3	0001000
201	1	1	000010000010000
201	1	2	100000100000100
201	1	3	001000001000001
201	2	1	000000000100000
201	2	2	000001000000000
201	2	3	010000000000010
201	3	1	000000000000000
201	3	2	000100000000000
201	3	3	000000000001000
201	4	1	000000000000000
201	4	2	000000000000000
201	4	3	000000010000000
202	1	1	0000100000100000100000100000100
202	1	2	0010000010000010000010000010000
202	1	3	1000001000001000001000001000001
202	2	1	0000000001000000000001000000000
202	2	2	0100000000000100000000000100000
202	2	3	0000010000000000010000000000010
202	3	1	0000000000000000000100000000000
202	3	2	0000000000010000000000000000000
202	3	3	0001000000000000000000000001000
202	4	1	0000000000000000000000000000000
202	4	2	0000000100000000000000000000000
202	4	3	0000000000000000000000010000000
202	5	1	0000000000000000000000000000000
202	5	2	0000000000000000000000000000000
202	5	3	0000000000000001000000000000000
203	1	1	000010000010000010000010000010000010000010000010000010000010000
203	1	2	100000100000100000100000100000100000100000100000100000100000100
203	1	3	001000001000001000001000001000001000001000001000001000001000001
203	2	1	000000000100000000000100000000000100000000000100000000000100000
203	2	2	000001000000000001000000000001000000000001000000000001000000000
203	2	3	010000000000010000000000010000000000010000000000010000000000010
203	3	1	000000000000000000010000000000000000000000010000000000000000000
203	3	2	000100000000000000000000000100000000000000000000000100000000000
203	3	3	000000000001000000000000000000000001000000000000000000000001000
203	4	1	000000000000000000000000000000000000000100000000000000000000000
203	4	2	000000000000000000000001000000000000000000000000000000000000000
203	4	3	000000010000000000000000000000000000000000000000000000010000000
203	5	1	000000000000000000000000000000000000000000000000000000000000000
203	5	2	000000000000000100000000000000000000000000000000000000000000000
203	5	3	000000000000000000000000000000000000000000000001000000000000000
203	6	1	000000000000000000000000000000000000000000000000000000000000000
203	6	2	000000000000000000000000000000000000000000000000000000000000000
203	6	3	000000000000000000000000000000010000000000000000000000000000000

Examining these patterns bit-by-bit for shift operations, power-functions, etc, may be a more practical way to decipher the winning patterns:

My goal is that the code should be able to find patterns, store the patterns in a dictionary and then consult a dictionary of patterns learned over time including patterns related to how to learn, match it up to the states based on how the variables change, and identify a recursive relation to solve for future games, not by using brute force, but by using the learned rules. As time goes on, we can add to the dictionary new algorithms and then use them to look for patterns. Therefore, this can be the basis for an extensible/self-learning framework.

Now, that we have captured the data states of the game, we can examine the game states as they relate to the number of variables. The engine for examining and learning the patterns can be the same engine that does the simulation, thus this can be a recursive self-learning organism feeding off of achieving goals within the constraints of rules. We should be able to apply this technique to any scenario, including complex ones by schematizing the scenario into a game-based schema. Since we can do that for Hanoi, although this is a very simple case, my thesis is that the same fundamental approach can be used for other scenarios, which may be much more complex.

I apologize that some of the code isn’t formatted nicely and wrapped strangely in some places. I used a handy add-in for inserting code (Insert Code from http://www.shahine.com/omar/InsertCodeForWindowsLiveWriter.aspx), but some of my code lines are too long and got truncated. Due to time constraints, I just went in and put hard returns where necessary in the blog post to make sure everything at least displays on the page.

If you feel like I left you hanging, it’s because I did. I will have more to post on this later after I figure out the recognition piece. Stay tuned…

Technorati Tags: AI,Artificial Intelligence,Simulation,SQL Server,Visual Studio .NET

Generating a List of Calendar Dates

Bob Leithiser — Mon, 06 Jul 2009 05:25:26 GMT

For today, here’s a simple trick. Ever need to get a list of all the calendar dates for a period? This is very simple using a user defined function with a table. Below is a simple version. I have a more complex version I am using for my application that filters based on another table containing holidays, etc.

-- =============================================
-- Author:        Bob Leithiser
-- Create date: 7/5/2009
-- Description:    Return a list of dates in table format for a specified date range
-- =============================================
CREATE FUNCTION [Util].[udf_GetCalendarDates]
(
    @StartDate DATE, 
    @EndDate DATE
)
RETURNS 
@CalendarDates TABLE 
(
    -- Add the column definitions for the TABLE variable here
    CalendarDate DATE
)
AS
BEGIN
    -- Fill the table variable with the rows for your result set
    DECLARE @CalendarDate DATE = @StartDate
    WHILE @CalendarDate  <= @EndDate
    BEGIN
        BEGIN
            INSERT INTO @CalendarDates (CalendarDate) VALUES (@CalendarDate)
        END
        SET @CalendarDate = DATEADD(DD,1,@CalendarDate)
    END
    RETURN 
END

GO


select * from util.udf_GetCalendarDates('20090101','20090130')

and now we get:

CalendarDate
2009-01-01
2009-01-02
2009-01-03
2009-01-04
2009-01-05
2009-01-06
2009-01-07
2009-01-08
2009-01-09
2009-01-10
2009-01-11
2009-01-12
2009-01-13
2009-01-14
2009-01-15
2009-01-16
2009-01-17
2009-01-18
2009-01-19
2009-01-20
2009-01-21
2009-01-22
2009-01-23
2009-01-24
2009-01-25
2009-01-26
2009-01-27
2009-01-28
2009-01-29
2009-01-30

Here’s an example where I found this useful. I needed to generate a list of holidays to avoid processing of days not containing any business activity data as part of an analytic/simulation application.

insert into Load.Holiday (ExchangeName,HolidayDate)
select '*', CalendarDate
from util.udf_GetCalendarDates('2007-05-01','2009-07-04')
where not exists
(select distinct marketdate from dbo.EquityHistory where MarketDate = CalendarDate )
and DATEPART(WEEKDAY,CalendarDate) NOT IN (1,7)

select * from Load.holiday

(20 row(s) affected)

ExchangeName    HolidayDate
*    2007-05-28
*    2007-07-04
*    2007-09-03
*    2007-11-22
*    2007-12-25
*    2008-01-01
*    2008-01-21
*    2008-02-18
*    2008-03-21
*    2008-05-26
*    2008-07-04
*    2008-09-01
*    2008-11-27
*    2008-12-25
*    2009-01-01
*    2009-01-19
*    2009-02-16
*    2009-04-10
*    2009-05-25
*    2009-07-03