Home of the Data Dude : PowerTools

Power Tools Installation Problem (error code 2738)

gertd — Sat, 12 Jul 2008 20:06:09 GMT

Some users have been reporting problems installing the VSDB Power Tools on Vista or Windows Server 2008 (applies to both 32 and 64 bit versions).

When they run the Power Tools installer they are confronted with the following mysterious error message:

Error code 2738 implies that the VBScript engine is not enabled. On some Vista and Windows Server 2008 installations the VBScript engine is disabled. The workaround is to enable the VBScript engine and run the setup again. After that you can disable the VBScript engine again if desired.

How to enable the VBScript engine:

Start an elevated command prompt.
Run the following command :
1. On a 32-bit OS run:
  regsvr32 %windir%\System32\vbscript.dll
2. On a 64-bit OS run:
  regsvr32 %windir%\SysWOW64\vbscript.dll

Now run the Power Tool installer from an elevated process.

How to disable the VBScript engine:

Start an elevated command prompt.
Run the following command :
1. On a 32-bit OS run:
  
  regsvr32 %windir%\System32\vbscript.dll /U
2. On a 64-bit OS run:
  regsvr32 %windir%\SysWOW64\vbscript.dll /U

I hope this helps, the next version of the Power Tools installer will no longer rely on VBScript.

-GertD

Command line SQL Static Code Analysis

gertd — Tue, 19 Feb 2008 01:06:05 GMT

We have had a lot of requests from users asking us to make T-SQL Static Code Analysis available at build time and from the command line. The latest Visual Studio Team System 2008 Database Edition Power Tool release, adds this ability to the product through a new MSBuild task. In this blog post I will go through the step that you need to follow to enable and use T-SQL Static Code Analysis from the command line.

Step 0: Install the Power Tools

You first need to install the latest Power Tools for Visual Studio Team System 2008 Database Edition.

Step 1: Create a new database project or open an existing one

We need a database project, either use an existing one, or simple create a test project, from example using the pubs database. That is what I will use for this example.

Step 2: Unload the project

We will need to add a entry to the project file to import the MSBuild .targets file for Power Tools. In order to do this you have to unload the project so that you can edit project file using the XML editor. Right click on the project node inside the Solution Explorer and choose "Unload Project".

Step 3: Import the Microsoft.VisualStudio.TeamSystem.Data.PowerTools.Tasks.targets file

The next step is to modify the project file to add an import statement to include the Microsoft.VisualStudio.TeamSystem.Data.PowerTools.Tasks.targets file which contains the MSBuild task for T-SQL Static Code Analysis. Right click on the unloaded project in Solution Explorer and choose "Edit <project name.dbproj>".

This will open the .dbproj file inside the Visual Studio XML editor. Now we need to add an import statement to load the .target file. If you search for "<Import" inside the project file you will find the regular import which loads the normal MSBuild tasks for the database project like the SqlBuildTask and the SqlDeployTask.

   1: <!--Import the settings-->

   2: <Import Project="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v9.0\TeamData\Microsoft.VisualStudio.TeamSystem.Data.Tasks.targets" />

We need to add a reference to the Power Tools targets file.

   1: <!--Import the settings-->

   2: <Import Project="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v9.0\TeamData\Microsoft.VisualStudio.TeamSystem.Data.Tasks.targets" />

   3: <Import Project="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v9.0\TeamData\Microsoft.VisualStudio.TeamSystem.Data.PowerTools.Tasks.targets" />

Step 4: Reload the project

After you made the project change, including the new targets file, it is time to reload the project. Right click on the unloaded project node inside Solution Explorer and choose "Reload Project". This will reload the project.

NOTE: Make sure you have no errors when loading the project, because this will be an indication that your edits were not correct.

Step 5: Start a "Visual Studio 2008 Command Prompt"

The next step is to run T-SQL Static Code Analysis from the command line, in order to do so we need a command prompt that has the PATH set correctly so we can call MSBuild.exe. Visual Studio creates a shortcut in the menu structure that will launch a command prompt with the correct settings.

When you started the command prompt navigate to the directory where you project file is located that you changed in the previous step.

Step 6: Run T-SQL Static Code Analysis from the command-line

From the command prompt run: msbuild pubs.dbproj /t:SqlAnalysis

NOTE: The results file is created in the output directory, which by default is the SQL directory

Step 7: Look at the results

When executing the project using the SqlAnalysis target, an XML file is created that contains the results, named StaticCodeAnalysis.Results.xml. This file contains the information that is normally displayed in the Visual Studio "Error List".

When you open the file in VS it will look like this:

   1: <?xml version="1.0" encoding="utf-16" standalone="no"?>

   2: <Problems>

   3:   <Problem>

   4:     <Rule>Microsoft.Design#SR0010</Rule>

   5:     <ProblemDescription>You should avoid using the following join syntaxes, which have been deprecated: Table1.Id = Table2.Id, Table1.Id *= Table2.Id, Table1.Id =* Table2.Id. You should use the newer syntax, which uses the JOIN keyword.</ProblemDescription>

   6:     <SourceFile>c:\work\pubs\Schema Objects\Stored Procedures\dbo.reptq2.proc.sql</SourceFile>

   7:     <Line>4</Line>

   8:     <Column>14</Column>

   9:     <Severity>Warning</Severity>

  10:   </Problem>

  11:   <Problem>

  12:     <Rule>Microsoft.Design#SR0010</Rule>

  13:     <ProblemDescription>You should avoid using the following join syntaxes, which have been deprecated: Table1.Id = Table2.Id, Table1.Id *= Table2.Id, Table1.Id =* Table2.Id. You should use the newer syntax, which uses the JOIN keyword.</ProblemDescription>

  14:     <SourceFile>c:\work\pubs\Schema Objects\Stored Procedures\dbo.reptq2.proc.sql</SourceFile>

  15:     <Line>4</Line>

  16:     <Column>23</Column>

  17:     <Severity>Warning</Severity>

  18:   </Problem>

  19:   <Problem>

  20:     <Rule>Microsoft.Design#SR0010</Rule>

  21:     <ProblemDescription>You should avoid using the following join syntaxes, which have been deprecated: Table1.Id = Table2.Id, Table1.Id *= Table2.Id, Table1.Id =* Table2.Id. You should use the newer syntax, which uses the JOIN keyword.</ProblemDescription>

  22:     <SourceFile>c:\work\pubs\Schema Objects\Views\dbo.titleview.view.sql</SourceFile>

  23:     <Line>4</Line>

  24:     <Column>15</Column>

  25:     <Severity>Warning</Severity>

  26:   </Problem>

  27:   <Problem>

  28:     <Rule>Microsoft.Design#SR0010</Rule>

  29:     <ProblemDescription>You should avoid using the following join syntaxes, which have been deprecated: Table1.Id = Table2.Id, Table1.Id *= Table2.Id, Table1.Id =* Table2.Id. You should use the newer syntax, which uses the JOIN keyword.</ProblemDescription>

  30:     <SourceFile>c:\work\pubs\Schema Objects\Views\dbo.titleview.view.sql</SourceFile>

  31:     <Line>4</Line>

  32:     <Column>23</Column>

  33:     <Severity>Warning</Severity>

  34:   </Problem>

  35:   <Problem>

  36:     <Rule>Microsoft.Performance#SR0005</Rule>

  37:     <ProblemDescription>You might cause a table scan when you use an expression that starts with "%" in the LIKE predicate. </ProblemDescription>

  38:     <SourceFile>c:\work\pubs\Schema Objects\Stored Procedures\dbo.reptq3.proc.sql</SourceFile>

  39:     <Line>6</Line>

  40:     <Column>73</Column>

  41:     <Severity>Warning</Severity>

  42:   </Problem>

  43:   <Problem>

  44:     <Rule>Microsoft.Performance#SR0007</Rule>

  45:     <ProblemDescription>You have specified a nullable column that is not wrapped with the ISNULL function. This practice can cause the final result to be evaluted as NULL for the predicate.</ProblemDescription>

  46:     <SourceFile>c:\work\pubs\Schema Objects\Stored Procedures\dbo.byroyalty.proc.sql</SourceFile>

  47:     <Line>4</Line>

  48:     <Column>7</Column>

  49:     <Severity>Warning</Severity>

  50:   </Problem>

  51:   <Problem>

  52:     <Rule>Microsoft.Performance#SR0007</Rule>

  53:     <ProblemDescription>You have specified a nullable column that is not wrapped with the ISNULL function. This practice can cause the final result to be evaluted as NULL for the predicate.</ProblemDescription>

  54:     <SourceFile>c:\work\pubs\Schema Objects\Stored Procedures\dbo.reptq3.proc.sql</SourceFile>

  55:     <Line>6</Line>

  56:     <Column>7</Column>

  57:     <Severity>Warning</Severity>

  58:   </Problem>

  59:   <Problem>

  60:     <Rule>Microsoft.Performance#SR0007</Rule>

  61:     <ProblemDescription>You have specified a nullable column that is not wrapped with the ISNULL function. This practice can cause the final result to be evaluted as NULL for the predicate.</ProblemDescription>

  62:     <SourceFile>c:\work\pubs\Schema Objects\Stored Procedures\dbo.reptq3.proc.sql</SourceFile>

  63:     <Line>6</Line>

  64:     <Column>27</Column>

  65:     <Severity>Warning</Severity>

  66:   </Problem>

  67: </Problems>

Step 8: Clean up the results

If you want to clean up the resulting file produced by the static code analysis run, simply run:

msbuild pubs.dbproj /t:ScaClean

This will remove the results file.

Conclusion:

Following the steps outlined before enable you to run T-SQL Static Code Analysis from the command line, you can use the same step to integrate this in to your Team Build environment, but you have to make sure that you install the Power Tools on the build server before you can use this.

-GertD
"DataDude" Group Engineering Manager

File Bound Generator

gertd — Sun, 17 Feb 2008 06:31:14 GMT

An other new addition to the Visual Studio Team System 2008 Database Edition - Power Tools release is the File Bound Generator. This data generator grovels through files inside a directory or optionally through its subdirectories, based on some search filter you provide and loads the content of the file in to a column value. The file content is either represented as text or as binary data. The generator is shared-instancing enabled, which means that the same file can be bound to multiple columns inside the same row while having to load it only once.

Let walk through an example which leverages the file bound generator to load data in to a table.

Step 0: Install the Power Tools

You first need to install the latest Power Tools for Visual Studio Team System 2008 Database Edition.

Step 1: Create a new project and add a table

The first step is to create a new database project, in this example I am using a SQL Server 2005 project named "testdb", since I want to leverage the NVARCHAR(MAX) and VARBINARY(MAX) data types, otherwise use NTEXT and IMAGE instead.

After having created the project add a new table which we will name [dbo].[files] with the following shape:

   1: CREATE TABLE [dbo].[files]

   2: (

   3:     [file_id]    int identity not null,

   4:     [file_name]  nvarchar(256) not null,

   5:     [file_text]  nvarchar(max) null,

   6:     [file_bin]   varbinary(max) null,

   7: );

After you created the project and the table, make sure that you build and deploy project.

Step 2: Create a new Data Generation Plan

Next add a new Data Generation Plan, by right clicking on the Data Generation Plan folder in Solution Explorer and choose: "Add... New Item... Data Generation Plan"

This will add a new Data Generation Plan to your project and automatically bind the columns of the tables inside your project to the default data generator.

Step 3: Create some test input

In order to create some test files we can load and also demonstrate how shared-instancing works I am using the following batch file:

   1: @rem CreateFiles.cmd

   2: @echo off

3:

   4: for /L %%I in (1, 1, 1000) do call :CREATE %%I

   5: goto END

6:

   7: :CREATE

   8: @echo filename%1.txt > filename%1.txt

9:

  10: :END

Create a folder, for example C:\TEST, create or copy the CreateFiles.cmd file in to the directory you just created and run the batch file. This will create a 1000 files with sequential names and the content of the file being the file name so we can use it for correlation purposes.

Step 4: Configure the File Bound Generator

The default configuration of the plan will look like this:

The next step is to assign file bound generator to the file_name, file_text and file_bin columns.

After you assigned the generator you need to set the Generator Output, by default the generator binds to the coercion type, so the file_name and file_text will bind to String Result since they are NVARCHAR columns, the file_bin column will by default bind to the Binary Result since its type is of VARBINARY.

We want the file_name column to bind to the name of the file we are importing, which means you need to change the Generator Output to File Name, the other columns are bound to the output that we need.

Bind the generator output to the columns, bind file_name to File Name, file_text to String Result and file_bin to Binary Result

The end result should look like this:

Next step is to configure generators for the 3 columns that we assigned to the file bound generator. If you want shared-instancing to work you need to make sure that the generator configuration of all 3 columns assigned to the file bound generator is identical.

Configuring the columns:

Step 5: Execute the Data Generation Plan

Now that we have configured the generators for all columns in the files table, we can generate some data, by executing the Data Generation Plan. Before we do this change the numbers of rows to be generated for the files table from 50 to 1000, to match the number of files we generated. Then hit F5, to execute the plan.

Step 6: Display the results

Once the executing is finished, open up the T-SQL editor and execute the following query.

   1: SELECT   *

   2: FROM     [dbo].[files]

   3: ORDER    BY [file_id]

The results:

If you click on the first binary entry for file_id = 1 you will find that the filename in the binary content matches the file_name and file_text.

Conclusion

The File Bound Generator allows you to load the content of files in to your database and the shared-instancing functionality makes sure that you can pull the absolute file path in to an other column in the same row when generating test data.

-GertD

Data Generation Wizard

gertd — Fri, 15 Feb 2008 10:34:06 GMT

One of the new features in the Visual Studio Team System 2008 Database Edition - Power Tools release is the addition of the Data Generation Wizard. The Data Generation Wizard is an item template wizard that allows you to create and configure a data generation plan by pointing it at an existing database, which has the same schema as the underlying database project, and configure each column inside each table with a sequential data bound generator. When finished you have a configured data generation plan that pulls all rows from the existing database in to your target database.

Lets step through the process.

Step 1: Add a new Data Generation plan using the Data Generation Wizard.

Inside an existing database project, select Add... New Item... and select the "Data Generation Wizard" option. In this example I am using the good old trusted pubs database as my sample database.

Step 2: Launch the Data Generation Wizard

Once you selected the Data Generation Wizard, you will get the main page of the wizard. Above you see the default state of the wizard.

Step 3: Select your source database

The next step is to select the database that you want to pull the data from, in this example I am pulling from the pubs sample database. If you have not configured a connection to the specific target database, you can create one using the New Connection... option or you can change existing connection definitions using the Edit Connection... option.

Once you select the database there are a couple of options you can select or set, lets walk through these:

Set the number of rows to generate...
This option will configure the "Rows to Insert" value of data generation plan, to exactly the number of rows that are inside the source database.
Replace the foreign key generator with values form the target database...
If you check this option the wizard will pull the foreign key values from the source database, instead of leveraging the foreign key data generator
Number of failed rows...
This option configures the error threshold for the number of failures to be allowed before terminating the population of a table
Number of failed tables...
This option configures the error threshold for the number of failures to be allowed before the terminating execution of the data generation plan

Step 4: Run the wizard

When you hit OK, you will see a Status window which will display the progress information of the creation and configuration of the data generation plan.

Below you see the status information from creating the data generation plan for the pubs database.

Retrieving data from pubs.
Data retrieval complete.
Starting merge of data generation plan with database schema.
Completed.
Resetting all target row counts to match target database
Setting table [dbo].[authors] to 23 rows
Setting table [dbo].[discounts] to 3 rows
Setting table [dbo].[employee] to 43 rows
Setting table [dbo].[jobs] to 14 rows
Setting table [dbo].[pub_info] to 8 rows
Setting table [dbo].[publishers] to 8 rows
Setting table [dbo].[roysched] to 86 rows
Setting table [dbo].[sales] to 21 rows
Setting table [dbo].[stores] to 6 rows
Setting table [dbo].[titleauthor] to 25 rows
Setting table [dbo].[titles] to 18 rows
Populating columns with Sequential databound generator
Configuring table dbo.authors to be databound
Configuring table dbo.discounts to be databound
Configuring table dbo.employee to be databound
Configuring table dbo.jobs to be databound
Configuring table dbo.pub_info to be databound
Configuring table dbo.publishers to be databound
Configuring table dbo.roysched to be databound
Configuring table dbo.sales to be databound
Configuring table dbo.stores to be databound
Configuring table dbo.titleauthor to be databound
Configuring table dbo.titles to be databound
Saving to file 'c:\work\pubs\Data Generation Plans\DataTransformPlan1.dgen'
Completed

As you can conclude from the status output the wizard configures the data generation plan in two phases. First it configures the row counts and secondly in configures the generators for each column inside each table. If you did not select the option to propagate row counts the first phase will be skipped.

Step 5: The resulting data generation plan

Now that the wizard is finished, lets look at the results. First we will look at the data generation plan. We will see that the row counts are configured for each respective table.

Step 6: The resulting column configuration

If we look at the column configurations we see that in our example the foreign keys are generated using the foreign key generator and the other values are pulled from the database. Note that we are using the sequential databound generator, which support multi-instancing. This means that columns values are matched per row, the regular databound generator does not have this ability to correlate column values within a single row!

Step 7: Running the generation plan

Now that we have a data generation plan, lets runs it. We created a clean database from our project named pubs2 and we are going to generate straight in to this.

If you are using the real pubs database you will find that this will fail. For a good reason, because the employee_insupd trigger inside the schema that will cause a rollback if the job id does not have a certain value. If you are using the default setting, the plan for the employee table will look like this:

The job id foreign key generator is not configured correctly to satisfy the employee_insupd trigger.

If you selected the option "Replace the foreign key generator with values form the target database..." you plan for the employee table looks like this:

Note that the job_id column is also pulled from the database, so when you used that option you will satisfy the trigger and the plan will just work!

Step 8: Showing the results

Now that we executed the plan, lets check the results, by opening up the T-SQL Editor and fire off a simple query "SELECT * FROM sales" to see if the content made it over.

Conclusion:

The Data Generation Wizard allows you to quickly configure a data generation plan that pulls all rows from an existing SQL Server database. The wizard will only allow you to pull information from an SQL Server database, because the schema of the objects inside the database has to match the table inside the database project.

We hope this new feature saves you time!

-GertD
"DataDude" Group Engineering Manager

Visual Studio Team System 2008 Database Edition Power Tools

gertd — Fri, 15 Feb 2008 08:59:06 GMT

The “DataDude” team is happy to announce the availability of the Power Tools for Visual Studio Team System 2008 Database Edition.

The Power Tools for 2008 contains all the functionality we shipped in 2005 plus:

Command line SQL Static Code Analysis execution through MSBuild, this was the biggest customer request, which is why I was holding the release. This enables SQL Static Code Analysis to be an integrated part of Team Build!
Data Generation Wizard; this is another customer request where we allow users to create a new data generation plan by pointing at an existing database, the plan will be fully configured by the wizard to pull all data from the database using the Sequential Databound Generator. This way users can use an existing data set and only override columns which impose risks because of for example privacy concerns and save about half a day or more of configuring a data generation plan from scratch, one column at the time.
File based data generator; this allows you to insert the content of files in to the database (works for string and binary, not for XML yet).
XML based data generator; this allows you to generate XML based on an XSD (the XSD has to be provided as file right now, and cannot be selected from database or inherited from the data type).
Unique Regular Expression generator; this adds the ability to generator unique values using the RegEx String generator.
Refactoring Command Generator has been made available as a MSBuild task for better project build integration so it can be made part of the pre-build and pre-deployment stages in the project. This allows users to automated the results of refactoring to some degree in to the project.
We added two new test conditions for Database Unit Tests
- ChecksumCondition – Which you can use to verify that the checksum of the data set returned by a database unit test matches the checksum of an expected data set.
- ExpectedSchemaTestCondition – Which you use to verify that the column names and data types of the returned data set match expected values.

Download page:
http://www.microsoft.com/downloads/details.aspx?FamilyID=73ba5038-8e37-4c8e-812b-db14ede2c354&displaylang=en

Installer download:
http://download.microsoft.com/download/f/b/8/fb8d1c0d-c0c4-4004-ab86-12396b2a3ee3/VSTSDB2008PT.msi

Documentation download:
http://download.microsoft.com/download/f/b/8/fb8d1c0d-c0c4-4004-ab86-12396b2a3ee3/Power Tools 2008.doc

Happy Valentine’s day!

-GertD

Data Bound Generator vs. Sequential Data Bound Generator

gertd — Wed, 05 Sep 2007 08:48:04 GMT

The Power Tools v1.0 release adds a new data generator, named the "Sequential Data Bound Generator". Since there already exists a data generator named the "Data Bound Generator" this yields the question what the difference is between the two is and when to use either one of them.

Data Bound Generator

The Data Bound Generator, takes the query you specified and uses it to fill a local dictionary. After query is execute, the results are used to fill a dictionary, the generator disconnects from the data source specified and randomly selects values from the dictionary and contributes the values to the column you bound the generator to.

This implies a couple of things:

The complete result set from the query has to fit in memory and is held in memory for the duration of the generation. (If this is not the case the data generator will fail with an out of memory error.)
The connection is only used to execute the query and fetch the results in to the dictionary, after that the connection is no longer used.
You cannot assign a distribution to the Data Bound Generator; the only pattern used to select from the dictionary is random. (This will get addressed in a later release, so you can specify a distribution to select from the dictionary using a statistical distribution pattern.)

Sequential Data Bound Generator

The Sequential Data Bound Generator, takes the query you specified, executes the query and fetches one row at the time and returns the value to the column bound to the generator.

This implies that:

Only a single row of the query is in memory at all times.
The connections used to execute the query and fetch the results is in use for the duration of the generation plan execution
The order in which values are returned are determined by the query statement, assuming you provides an ORDER BY clause in the query.

Both providers can connect to the same providers and are based on the same query execution code, only the way results are handled is different between the two.

I hope this explains the differences between the two providers and makes it easier to determine when to use which data generator.

-GertD

Channel 9 video Covering Power Tools & Service Release 1 Functionality

gertd — Fri, 17 Aug 2007 07:40:05 GMT

Before I left for vacation, I did a quick video recording for Channel 9 together with Brian Keller showing off the new functionality in the Visual Studio Team Edition for Database Professionals Power Tools and Service Release 1.

You can find the video at this location http://channel9.msdn.com/showpost.aspx?postid=334261

Brian also released a video he did with Sachin Rekhi on "Offline Database Development", which you can find here http://channel9.msdn.com/Showpost.aspx?postid=334518

Have fun,

-GertD

Power Tools Download Link

gertd — Sat, 11 Aug 2007 17:21:51 GMT

They are here!

Dependency Viewer
Refactoring
- Move Schema
- Expand Wildcard
- Fully Quality Name
- Refactor in to strongly typed DataSet definitions
- Refactor Command Generator
Data Generation
- Sequential Data Bound Generator
- Editors for the Data Bound Generator, Sequential Data Bound Generator and RegEx String Generator to make configuration easier
- The RegEx editor also tries to interpret your CHECK CONSTRAINTs and create a matching RegEx expression that you can use to generate data values that match the constraint definition
- The RegEx editor can also be used for interactively defining and testing RegEx expressions and evaluate the output visually, which makes it a lot easier to create the right RegEx expression for your value domain.
MSBuild Tasks
- SqlSchemaCompareTask; allows you to compare schemas between two database from the command line using MSBuild.
- SqlDataCompareTask; allows you to compare the content of tables within two databases from the command line using MSBuild.
T-SQL Static Code Analysis
Miscellaneous tools
- SQL script pre-processor command-line utility, which will expand all SQLCMD includes and variable definitions (sqlspp.exe)
Schema Manager API

See http://blogs.msdn.com/gertd/archive/2007/08/07/it-is-august-6th.aspx for a more detailed explanation.

The Visual Studio Team Edition for Database Professionals first Power Tools release can be downloaded from the following location:

Power Tools download: http://go.microsoft.com/fwlink/?LinkId=88852

The Power Tools require that you have Service Release 1 installed, which you can download from the following location:

Service Release 1 download: http://go.microsoft.com/fwlink/?LinkId=88853

I will be offline for 3 weeks and I will pick up the blogging pace with lots of new posts when I am back after Labor day.

Later,
-GertD

What is in the Power Tools v1.0 release?

gertd — Tue, 07 Aug 2007 10:32:00 GMT

For those of you who have been paying close attention, you know that August 6th is the date I promised the first Power Tools to be released. So where are they?

We are not there yet, we released the first release candidate today, I need to finish two more threat models and then we are ready to rock & roll, so it will be this week, better be because I am going on vacation for 3 weeks and I promised my boss it will be done before I leave. So please be patient a little bit longer, it will not be long before you can enjoy these great new tools.

Which Tools?

So here is a list of the new features and functionality we are adding, most of these you have been asking us for, so here it is, no need to wait for a completely new release.

Dependency Viewer
Refactoring
- Move Schema
- Expand Wildcard
- Fully Quality Name
- Refactor in to strongly typed DataSet definitions
- Refactor Command Generator
Data Generation
- Sequential Data Bound Generator
- Editors for the Data Bound Generator, Sequential Data Bound Generator and RegEx String Generator to make configuration easier
- The RegEx editor also tries to interpret your CHECK CONSTRAINTs and create a matching RegEx expression that you can use to generate data values that match the constraint definition
- The RegEx editor can also be used for interactively defining and testing RegEx expressions and evaluate the output visually, which makes it a lot easier to create the right RegEx expression for your value domain.
MSBuild Tasks
- SqlSchemaCompareTask; allows you to compare schemas between two database from the command line using MSBuild.
- SqlDataCompareTask; allows you to compare the content of tables within two databases from the command line using MSBuild.
T-SQL Static Code Analysis
Miscellaneous tools
- SQL script pre-processor command-line utility, which will expand all SQLCMD includes and variable definitions (sqlspp.exe)
Schema Manager API

A More Detailed Look

Dependency Viewer

The Dependency Viewer is a new tool window that can be accesses by right clicking on a node inside Schema View and choosing the "View Dependencies" option. This will bring up the Dependency Viewer tool window. The dependency viewer shows the internal graph/dependency chain maintained internally, which is build up when we load the project and parse all the schema objects inside the project.

The viewer shows which objects are "Referenced By" and "References" the object you selected. You can also drag objects in to the viewer, or continue to right click "View Dependencies" on objects, the viewer remembers which objects you selected. Right clicking on an object in the viewer will bring up an context menu that allows you to select the object inside "Solution Explorer", "Schema View" or open the object inside the T-SQL editor by using the "Open" command. The "Remove" command will remove the object selected from the viewer (this will not remove it from the project), "Remove All" will remove all objects from the viewer. Refresh re-queries the internal schema manager for any updates.

Refactoring

Refactoring is a huge feature of VSDBPro! It really shows the abilities of the product and the power gained by understanding the relationships between all schema objects and the definitions of each schema object. When we shipped we purposely limited ourselves to rename refactoring, instead of adding more we added an extensibility mechanism that allows us to add new refactoring types and targets out-of-band, through Power Tools or other means.

So here is the first set of additions to refactoring:

New refactoring types

Move Schema; this allows you to move an object between existing schemas
Wildcard Expansion; this will take any SELECT * and convert it in to a properly defined column list, if the table or view is referenced through an alias we will use the alias the properly expand the column list otherwise we propend the fully qualified table or view name
Fully Qualify Name; this will fully qualify each object reference, this disambiguates the your query statements completely

A new refactoring target

We are adding a new refactoring target, namely strongly type DataSet definitions. If you have a solution (.SLN) which contains a database project (.DBPROJ) and a VB.NET or C# project which itself contains a strongly type DataSet definition (which is an XSD at the end of the day), we will refactor in to the strongly type DataSet definition if the refactoring touches an object used in the DataSet.

In this example we have a solution with a C# project that references the Customers table, however the Last Name column contains a space which is inconsistent so we will use rename refactoring to remove the space.

If the DataSet designer is open, refactoring will cause the following dialog to appear, notifying you we are going to evaluate the content of the XSD.

The refactoring preview windows will show you the following results

Note that the DataSet is not by default included as a refactoring target, you need to opt-in to propagate the changes in to the DataSet. When you select the DataSet we will only update the XSD, not your C# or VB.NET code, so your application will recompile and work, but for example text labels inside your code might be mismatched. Anyhow this is a great preview of where we are going with the product and our desire to bridge the gaps between the application and data tier developers.

Refactoring Command Generator

You might know that every time you perform a refactoring operation we create a log file, which describes the type of operation that you performed, on which object, what the exact change was and which other objects were affected. The log files live in the "Refactoring Logs" directory in your project directory, if you select "Show All Files" in "Solution Explore" you will see them as well.

We added a tool to use these log files and extract useful information out of them. For example of you renamed an object the build system will not create a rename operation, since in time it can not distinct a DROP/ADD column of the same type from a renamed object. Using the refactoring logs we preserved the intend of the user. The Refactoring Command Generator tool will take these logs and create deployment statement that reflect the operation of the user. So a rename of a table or column will be reflect through sp_rename, a move schema will be reflected through a ALTER SCHEMA TRANFER.

The Refactor Command Generator is available through the standard refactoring context menu and through the Data->Refactoring menu. There is also a command line version named RefactorCmdGen.exe which lives in the %ProgramFiles%\Microsoft Visual Studio 8\DBPro directory.

Data Generation

We added a new generator named the "Sequential Data Bound Generator", the big difference with the Data Bound Generator is that, this one only has a single row in memory at a given time and rows are dispatched in the order they are retrieved from the query results. The Data Bound Generator uses the query to fill a dictionary, holds this in memory for the duration of the generation and randomly select values from the dictionary.

To make configuration of the hardest generators a bit easier we added editors for the Data Bound, Sequential Data Bound and RegEx String generators. The editor manifest themselves by adding ... at the end of the row in the Column Details window.

The data bound generator share the same editor, which allows you to select the connection and enter the query (sorry no query builder yet).

The RegEx editor is more advanced.

It shows the expression and creates sample output, so you can validate the results. You can also select from a set of predefined expressions and add your own. The editor is also capable of interpreting CHECK CONSTAINTS and propose a matching expression that meets the value domain definition defined by the constraint.

MSBuild Tasks

We added two MSBuild tasks to perform schema and data comparisons from the command line.

SqlSchemaCompareTask; allows you to compare schemas between two databases.
SqlDataCompareTask; allows you to compare the content of tables and views within two databases .

T-SQL Static Code Analysis

One of the biggest features we are adding is T-SQL Static Code Analysis, like refactoring it build on top of our ability that we fully understand all object definitions and relationships, which allows us to perform cross cutting analysis over your schema. For example we can detect that a column is defined as an INT, a variable of a stored procedure as REAL and that you are assigning the column value to the variable, therefore incurring the cost of an implicit conversion. Rule: "SR0014: Maintain data type compatibility" will warn you for these kind of occurrences.

You can configure the rules used in the new project property page named "Static Analysis", you can determine which rules to include and if you want to threat them as warnings or errors. You can suppress warnings/errors by right clicking on the line in the Error List, and choosing Suppress Message(s), this will add the objects in to an XML file named StaticCodeAnalysis.SuppressMessages.xml which is added to your project when you add the first suppression. In order to run the static code analysis, you choose Data->Static Code Analysis->Run. This will run each rule selected and report the results in the Error List.

In the current implementation we do not allow you to write your own rules, in the Rosario release this will be enabled, for now send me email through the link in this blog to make your desired rules known. We will update them frequently, right now we have about 82 rules defined, you find the first 15 in this release, we have about 15 more to come to you shortly. An other limitation is that you can only run static code analysis from the IDE, there is no command line or MSBuild equivalent, this will also be solved in the Rosario release.

Miscellaneous tools

In order to use build scripts generated with VSDBPro with tools that do not understand SQLCMD variables and or include file, we added a simple command line tool that pre-processes SQL scripts and expand all includes (:r) and SQLCMD variable references. (sqlspp.exe)

Schema Manager API

The last part of the Power Tools is a small but very powerful API that provides access to the internals of the system, called the Schema Manager. This API can be used to add, update, delete schema objects to the system, to enumerate schema objects and files. This API is used by Computer Associates (CA) to integrate ERWin 7.2 with VSDBPro so you can make changes in ERWin and store them schema definition inside a database project. See http://ca.com/us/content/campaign.aspx?cid=144449 for more details on the ERWin integration.

That is it!

I hope there is something in here you like, please continue to send me your feedback, good or bad.

Thanks,
-GertD
"DataDude" Software Architect