The Zen of Database Development

Katmai: Variable Initialization and Assignment Operators

2007-08-23T20:51:00Z

It’s again.

Today I’m going to go over two new enhancements related to TSQL variables and how they acquire their values.

TSQL Variable Declaration and Initialization

In SQL 2008, the syntax by which TSQL variables may be declared has been enhanced. It is now possible to initialize variables during the DECLARE statement. This initialization uses virtually the same syntax as initializing variables using a SET statement. So while in SQL 2005, initialization might look like

DECLARE @i INT, @j INT, @k INT
SET @i = 10
SET @j = 11
SET @k = SELECT MIN(OrderID) from Orders

In Katmai, this can be all done in one statement

DECLARE @i INT = 10, @j INT = 11, @k INT = (SELECT MIN(OrderID) from Orders)

Katmai does not allow you to use a variable in the initialization of another variable in the same DECLARE statement. Because there is no enforced order of precedence in the processing, this syntax is not allowed:

DECLARE @i INT = 10, @j INT = 11, @k INT = (SELECT MIN(OrderID) + @i from Orders)

However, this is:

DECLARE @i INT = 10
DECLARE @j INT = @i + 1
DECLARE @k INT = (SELECT MIN(OrderID) + @i from Orders)

Note that Katmai still does not permit mixing Scalar declarations (those that declare a TSQL variable capable of holding a single value, such as above) with either Table Variable declarations or Cursor declarations.

Compound Assignment Operators

Katmai has added support for compound assignment operators to any statement where an assignment occurs on the right hand side of an expression. For those who may not be familiar with the term, Compound Assignment operators are used to update an existing value in place. For instance, the statement:

SET @result = @result + 1

Is functionally equivalent to the compound assignment statement:

SET @result += 1

The list of compound assignment operations that Katmai supports will be very familiar to anyone used to working in C# or other procedural languages.

Description	Syntax
Add and assign	+=
Subtract and assign	-=
Multiply and assign	*=
Divide and assign	/=
Modulo and assign	%=
Bitwise & and assign	&=
Bitwise \| and assign	\|=
Bitwise xor and assign	^=

This includes the ability to assign values to TSQL Variables using SET, as well as set values in UPDATE statements and set variables in SELECT statements.

Some examples:

SET @total += dbo.calculateInterest(@amount);

UPDATE Accounts
SET balance -= 200
WHERE accountId = @account_ID

UPDATE Accounts
SET @total = balance += dbo.calculateInterest(@amount)
WHERE accountId = @account_ID

These are some pretty simple features to experiment with. Next week, we'll get more ambitious, and I'll discuss the new Grouping Sets enhancement.

Until then!

Best Practices : The Truth of the Schema is In The Database Project

2007-08-22T19:23:00Z

Wednesday is at ZoDD. This is the day for focusing on the Whys behind the Hows of using Visual Studio Team System for Database Professionals, and for trying to provide some direction for dealing with different issues regarding the Application Development Lifecycle, as it relates to our product.

I’m going to start this series out with a brief post on the basic principle behind DBPro – isolated, off-line development.

It is no accident that DBPro is part of the Visual Studio family. Its emphasis has been on SQL Development from its inception. This does not mean we have no interest in the operational aspects of a SQL environment, but that our first focus has been to serve the Database Development community.

In the world of databases, there are two fundamental “truths”. There is the Truth for Data, and the Truth for Schema. In the majority of SQL installations today, the source of both these truths resides in one location: the production database. If you really want to know what the structure is for that table, there’s only one place to go to find out: the production database. If you want to understand what a particular Stored Procedure is doing, you have to get access to the Production Database to make sure that you are seeing what is really going on. And good luck if you have more than one instance of the Production Database – now you have to make sure they’re always structurally in sync, and if one is different from another, you have to figure out which one is correct, and then figure out how to get the other(s) into alignment. It can make for some long nights!

So one of the fundamental principles we started with was to move the Truth for Schema off of the Production Database. The database remains the Truth for Data, but we take the database schema – and by this we mean both the structural models (Tables, Views, etc) and the programmability objects (Stored Procedures, Functions, etc.) – reverse engineer them out of the Production Database into individual script files, load them into a Visual Studio Database Project, and put that project under Source Control. Now you have your schema in a managed environment, and using DBPro, you can coerce your Production Databases to conform to a centralized Schema definition. You have all the benefits of Source Control in a Development environment: Labeling, Versioning, History, rollback to previous versions, and the ability to manage conflicting schema changes during checkin. Moreover, now each developer has everything he needs to create a “Sandbox” environment for development and testing, and assure that all changes are correct and consistent prior to ever exposing the Production Database to them. We’ll be going over how each of these pieces fit into the overall Database Development Lifecycle in upcoming posts.

In DBPro, the Database Project is The Truth for Schema, and our entire product is focused on providing support for this project, and effectively managing the Schema of the Production Database from there.

Next Time : Roles in the Database Development Lifecycle

Getting DataDude, or What Do I Install Next?

2007-08-20T20:12:13Z

Monday is at ZoDD. Monday’s we’ll be discussing some subject that relates directly to Visual Studio Team System for Database Professionals (aka DBPro) – a feature or a new Power Tool or a new piece of available functionality.

Today I’m going to start off by addressing something that I get a lot of questions about – namely, where does DBPro fit into the overall installation scheme of Visual Studio, and what is the correct sequence of events to get the whole thing installed properly.

First off, it is important to understand that DBPro is a part of the Visual Studio Team System (VSTS) suite. This is distinct from Visual Studio Professional, which is oriented towards the individual developer. Team System is an integrated set of Visual Studio products focused on the ALC (Application Life Cycle), and DBPro contributes the database support portion of that integration. So a bit of the functionality that makes DBPro powerful is actually derived from features already in the larger Team System suite of which it is a part. That isn’t to say that you can’t use DBPro on top of Visual Studio Professional – but you do lose some of the integration features. DBPro on top of VSPro will be a subject of an upcoming post.

Anyway, DBPro released its first version in December of 2006, and this drop was out-of-band from the rest of Visual Studio Team Suite. As a result, the Version 1 bits were made available as an add-on to the existing VSTS installation. So, to get the whole shebang, you need to first have VSTS, and then install DBPro on top of that installation. The DBPro Version 1 bits can be found here. If you want to just try this stuff out, then you can download a trial version of VSTS (which you can find here), install it, and then download and install the standard DBPro installation described above on top of that. In this case, when the VSTS trial expires after 180 days, the DBPro bits will go with it.

So that gets you the initial installation. However, since December, the DataDude team has made a bunch of improvements to the original functionality, and on July 23^rd, we released our first service release. This release came out as a patch to DBPro version 1, which means that you must first have the standard drop described above already installed. But before you can install the new patch, you must first be sure that you have the first Service Pack for VSTS installed, which can be found here. Note that if you are installing on top of the VSTS trial download, you must make sure you add this Service Pack, as the trial does not come with the pack pre-installed.

Once you have VSTS SP1 installed, you can add the DBPro Service Release 1 to your installation. DBPro SR1 can be found here.

There’s one more, optional, step. The DataDude team has a commitment to bringing a stream of continuous updates your way, and part of how we accomplish this is through a series of Power Tools. Power Tools are add-in features to DBPro which may be installed selectively, and are not required for core DBPro functionality. We make these tools available as previews to upcoming features, or as a way of testing the waters for the usefulness of potential DBPro enhancements, or just to provide useful new features without having to wait for the rigors of official release management. Note that these Power Tools are not covered by the normal support channels, so if you have problems or issues with them, you need to bring them up in our Forums. We just had our first official Power Tools drop last week, and they can be installed from here. I’ll cover some of the Power Tool functionality in upcoming posts, and you can read about them on Gert Draper’s Blog.

It’s worth noting that DBPro requires a local instance of SQL Server 2005 running to use as our validation database. We’ll run on top of SQL Express, so if you install VSTS, the default instance should be fine. If you have a local instance of SQL 2005 Developer Edition, we recommend you use that.

So, in summary, to get the full range of DBPro, here’s the sequence of events:

· Visual Studio Team System full installation (link to trial)

· Visual Studio Team System for Database Professionals, version 1 (link)

· Visual Studio Team System, Service Pack 1 (link)

· Visual Studio Team System for Database Professionals, Service Release 1 (link)

· Visual Studio Team System for Database Professionals, Power Tools Release 1 (link ,optional)

And finally, as a side note, when Visual Studio Team System 2008, code-named Orcas, ships, DBPro will be finally integrated into the main VSTS installation package, and you won’t have to manually add us to the installation any more.

Hope this clarifies some of the questions I’ve heard. Next week, I’ll talk about what DBPro looks like when you drop it on top of a Visual Studio Professional installation. Until then!

Samples: Registering Extensions for VSTS.DBPro

2007-08-17T22:35:55Z

Friday is at ZoDD, where I’ll try to bring you some working code samples that leverage features and functionality you can find in the currently released version of Visual Studio Team System 2005 for Database Professionals.

Today begins a series of posts that will deliver on a long-standing, but all-too neglected promise I’ve been making at the various conferences and presentations I’ve done over the last couple of months – namely, providing working samples, with explanations, for two key scenarios that highlight DBPro’s customization capabilities: a Custom Data Generator and a Custom Unit Test Condition.

But this first entry is going to have a very specific focus – how to build and register these customizations in Visual Studio. It may seem a little out of sequence, but both of these customization projects will end up using the steps I’m about to describe, so I figured I’d cover them up front, and get them out of the way – these are, at least to me, the less interesting parts of the projects.

Our customizations are fundamentally Visual Studio add-ons. This means that they need to be registered with Visual Studio as a whole, and if you’ve built other VS add-ons, then this is all going to seem boringly familiar. But for those of you to whom this seems a little more esoteric, follow along, and we’ll have you setting these babies up in no time.

These steps assume you have created a Visual Studio Class Library that you want to register as an extension in Visual Studio for use in DBPro. There are a couple of things you have to do to finish up the output and get it into VS.

First off, you have to build a signed DLL. To do this, open up the Project Properties window, select the Signing Tab, choose Sign the Assembly. You may choose to either specify and existing key, or just create a new one. Unless you have a specific key you want to use, just go ahead and select <New...> Now you can Build the DLL from the Build menu.

Once you have Built the DLL, you need to get it into the Global Assembly Cache, or GAC, because that’s where we expect to find our extensions. If you don’t already know, the GAC is generally under your WINDOWS directory (or WINNT if you’re on a server), as a directory called assembly. In order to get the assembly into the correct directory and registered, you use the gacutil command from a command prompt. If you use a Visual Studio command prompt by going to the Visual Studio Tools menu from the Start menu, the gacutil will already be in your path. If you just use a normal command prompt, you’ll want to add it to your path. It can generally be found at [Program Files]\Microsoft Visual Studio 8\SDK\v2.0\Bin.

Once you have both the gacutil and your freshly brewed DLL in your path, you can push it into the GAC using:

GACUTIL /if NameOfCustomExtension.DLL

Where NameOfCustomExtension is, of course, the name of the custom extension you just built. The /if will install it, and if it is already out there, override it.

Now that it’s in the GAC, we need something from the installed version. If you navigate to the GAC using Windows Explorer, the entries in the assembly directory will have a column called Public Key Token. Find the assembly that you just GAC’d, and copy its public key token – you’ll need it in a minute.

We’re down to the wire here. The last step we need to perform is to tell Visual Studio about our new assembly, and that it can be used in DBPro. We do this by adding an XML file with an extension of extensions.xml into the specific location where DBPro installs: by default, this is [Program Files]\Microsoft Visual Studio 8\DBPro.

The contents of the extensions file are fairly standard, fairly specific, and not very complicated. Here are the entries you’ll need for each individual library you publish. Create a document called NameOfCustomExtension.extensions.xml and open it up with your favorite XML editor – notepad will do just fine. Copy and paste the block below into your new doc. I’ve highlighted the portions that you’ll need to change for each individual DLL.

<?xml version="1.0" encoding="utf-8"?>

</extensions>

Most of these changes should be pretty straightforward – the bulk of all this is to point to the correct namespace for DBPro extensions. Obviously, the NameOfCustomExtension is the name of your DLL, just as above. Replace the Public Key Token with the one you copied from the GAC entry. As to the extension type, well, you could have several extensions in a single Assembly. You would need one of these entries for each of those extensions. We’ll see in detail how this all ties together when we build our first extension next week.

We’re almost done. The only remaining step is to make sure you shut down Visual Studio and reload it before you can expect to see your new features. Visual Studio loads this stuff when it first fires up. And, if everything went as planned, we should be all set!

We’ll be referencing this post in future customization posts, so keep it in mind. I’m still debating whether to start off with a custom Unit Test Condition or a custom Data Generator. In any case, we’ll step through the process and code of building one or the other next Friday.

See ya then!

Katmai: Looking into Row Constructors

2007-08-16T19:58:00Z

Thursday is here at ZoDD, and today I’m going to be discussing the first in a series of posts covering features that show up in SQL Server 2008, using the latest Community Technology Preview, CTP4. This morning’s subject will be the new functionality they are calling Table Value Constructors, and sometimes, perhaps more clearly, Row Constructors. Personally, I hope the SQL Team decides to go with the “Row Constructors” term, because the other sounds a little too much like Table-valued Parameters - which is another, completely different feature I’ll be covering in another post at a later date. Anyway, Row constructors, as I’ll call them for now, are a new TSQL construct in SQL Server 2008 that leverages some familiar syntax from the INSERT Statement to expose some pretty handy new functionality.

So, if you’re familiar with INSERT syntax, you’ll remember that the way to insert individual values into an inserted row is to use the VALUES clause. (This is, of course, the alternative to inserting valves using a SELECT statement - but that’s a different subject). In Yukon, (SQL server 2005) a VALUES statement looks something like this:

INSERT INTO Contacts (Name, Age, SalesId)

VALUES(‘John Doe’,25,5)

If you want to add additional values, then you would use additional INSERT statements.

INSERT INTO Contacts (Name, Age, SalesId)

VALUES (‘Jane Doe’, 36,6)

INSERT INTO Contacts (Name, Age, SalesId)

VALUES (‘Peter Doe’, 49, 7)

But now, in Katmai, these multiple sets of values may all be assigned within a single VALUES clause. This syntax opens up this clause for some very table-like behavior. For instance:

INSERT INTO Contacts (Name, Age, SalesId)

VALUES(‘John Doe’,25,5), (‘Jane Doe’, 36,6), (‘Peter Doe’, 49,7)

I’ve had a lot of customers in the past who load up domain value tables through scripts that consist primarily of a host of separate INSERT statements. This new VALUES clause syntax will make it much more convenient to load these sorts of tables. It’s also worth noting the impact this has on the nature of implicit transactions used in this context. Instead of having to explicitly define a transaction context that wraps a whole series of discrete INSERT statements, the entire set of valves is bounded by the single transaction, implicit in the single INSERT statement itself.

But wait-there’s more!

Now that we have established how to use these Row Constructors in the INSERT statement, Katmai expands their usage to allow them to be used as a more general table source.

SELECT *

FROM (VALUES(‘John Doe’,25,5),

(‘Jane Doe’, 36,6),

(‘Peter Doe’, 49,7))

Here, the VALUES statement is parenthesized and used just like it was a logical table. This has a lot of implications. There’s a whole set of TSQL Statements that are able to use this VALUES clause in places where we used to find a table or a subquery.

OK, so I lied. If you actually execute that line in Katmai, it will fail with a syntax error. That’s because there is another piece of syntax enhancement that needs to be in place in order for the parser to make sense of this. Here’s the statement that actually does work:

SELECT *

FROM (VALUES(‘John Doe’,25,5),

(‘Jane Doe’, 36,6),

(‘Peter Doe’, 49,7)) psuedoTable(pName, pAge, pSalesID)

Check out the new aliasing syntax. Now you can not only specify a name for the “table”, but you can name all of the columns in the table as well. This’ll come into heavy play later when we cover INSERT OVER DML. For now, suffice it to say that this effectively let’s you set up tables in script. Note that this aliasing is not necessary for use in the INSERT statement.

This can get more sophisticated:

SELECT *

FROM Employees emp

JOIN (VALUES(‘John Doe’,25,5),

(‘Jane Doe’, 36,6),

(‘Peter Doe’, 49,7)) psuedoTable(pName, pAge, pSalesID)

ON emp.EmployeeID = pseudoTable.pSalesID

Think about how this can affect your application!

And if that’s still not enough, Katmai restores a capability found in SQL 7.0 that was abandoned in Shiloh and Yukon – the ability to include scalar (as in, single value, not multi-column) subqueries as elements in a Row Constructor. Each of the these subqueries can only return one value – useful for extracting minimums or maximums from a table. And again, this same VALUES clause can be used in place of a table in SELECT syntax.

INSERT INTO Contacts (Name, Age, SalesId)

VALUES ((SELECT Top 1 Name FROM foo),

(SELECT Top 1 Age FROM foo),

(SELECT Top 1 fooId from foo))

Note that you cannot mix scalar subqueries with straight values in the same VALUES clause. So this would be invalid:

VALUES ((SELECT Top 1 Name FROM foo), 35,7)

This is just one of many features in the new Katmai CTP. If you haven’t tried it out yet, you can download the public July Community Technology Preview, version 4, from here. The SQL Team wants your feedback. Here at DataDude, we’re deep into figuring out what all these new features are going to mean to DBPro. Check it out!

Next Week: We’ll cover Variable Initialization and Assignment Operators.

Return of the Blogging Dead

2007-08-16T05:03:45Z

So, embarassingly enough, this blog has been devoid of activity for some time now. Well, let's see if we can't catch up on some of this.

For those of you who have been patiently (and not so patiently) waiting for my downloads for Custom Data Generators and Custom Unit Test Conditions, stay tuned! That series is (finally) coming right up.

Check back in tomorrow for the first in a series of entries about the new features in Katmai ... eh, SQL Server 2008 ... based on my investigations into the new Katmai CTP 4!

See ya then!

DBPro - Public CTP for Service Release 1 released.

2007-05-07T20:25:00Z

Ok, let's do the big news first...

The One and Only Public CTP for DB Pro's Service Release 1 - has gone live!

Since we just RTM'd version 1 in December, this is a public CTP of our first Service Release. But it does more than address a few bugs. It has some cool new features that we just didn't have time to get into our original release.

First of all, we now have extensive support for three and four part names in DB Pro. By using a slick custom referencing system, we are able to resolve three and four part names between multiple projects - even if you don't have all the projects in your current solution. We do this by generating a special assembly-like metadata file that contains the necessary information to support the resolution, which can be included as a reference in your project.

Second, we expanded support for filegroups and the files that make them up. Specifically, in version 1 of DB Pro, Filegroups were handled as a special type of schema object, with their own .filegroups.sql script file. Now, filegroups and their associated files are a part of the database project properties page, and may be defined and referenced in a new UI.

We also added additional support of SQLCmd variables - the ones you define with SETVAR in a SQLCmd script - allowing you to use them for increased flexibility in defining names and deployments. They are also a crucial part of the implementation of the first two features.

Pointers to further documentation and links can be found on Gert's blog here. In addition, here is a link to the download itself. Note that it requires Visual Studio Team System SP1 to work properly - check out the Release Notes.

We're anticipating going to RTM with this Service Release in early June - so this CTP is just a taste. Try it out, and let us know your feedback on the forums!

Datadude and the Text Filegroup

2007-01-28T02:16:00Z

So, we had an interesting problem crop up from a European customer, and investigating it uncovered both a little known, um... feature in SQL Server, and a bit of a twist in our handling of it.

The customer had a table that he was Reverse Engineering from his source database into a DBPro project. The table contained nothing but integer columns. However, when he completed the import, the Create Table script that we generated had an entry at the bottom, after the ON [PRIMARY] clause, that said TEXTIMAGE_ON [PRIMARY].

If you've never run into this before, this clause is used when SQL finds there is some sort of Large Object (LOB) in your table: in SQL 2005, this could be an XML column, or an NVARCHAR(MAX), or any of a number of things that lack a predefined size. SQL allows you to put these Large Objects somewhere other than the main table, and even allows you to put them into a different filegroup. Therefore, every time it finds a LOB in a table, it makes an entry into a column in sys.tables called lob_dataspaces_id, recording what filegroup the column is going to live in. Even if it is just going to stay in the same space as the rest of the table.

You can easily see this filegroup if you right-click and select the Properties of a table in SQL Server Management Studio. At the very bottom of the Property dialog is an entry called Text Filegroup. If there is an LOB in the table, this will be populated with the name of a filegroup. If it is blank, then there are no LOBs in the table.

Datadude uses the existence of this entry to control whether or not we append the TEXTIMAGE_ON clause onto the end of our generated Create Table scripts. So what happened to this customer?

Well, it turns out that if a table used to have a LOB in it, then the Text Filegroup is preserved. Deleting the LOB column does not zero the entry out of the lob_dataspaces_id column in sys.table. Imagine that. So when we go to get the table, we return the information about the Text Filegroup, and automatically append the clause onto the Create Table statement.

The effect of this is an error when the Create Table statement gets validated inside the DBPro project:

Msg 1709, Level 16, State 1, Line 1

Cannot use TEXTIMAGE_ON when a table has no text, ntext, image, varchar(max), nvarchar(max), varbinary(max), xml or large user defined type columns.

And now your script is in an error state. And sure enough, the customer verified that their table did indeed previously have a LOB in it.

This is easy enough to fix. All you have to do is delete the offending clause in the editor, save the script, and the problem disappears. The only other wrinkle is if you are intending to deploy back to the original SQL Server database - the one with the lob_dataspaces_id still defined. During our Build and Deploy process, Datadude will see the change in the table script, and conclude that the table needs to be rebuilt. On deployment, it will create a new table with the same schema, move all the data from the original, and then swap the names around. Being a new table, this one will no longer have the old, residual Text Filegroup setting. Problem gone.

Of course, this isn't our final resolution - we never really intended this to work this way. Since SSMS is smart enough to figure out not to script out the TEXTIMAGE_ON clause in their Create Table scripts, we'll figure out how to suppress it in ours as well. But when we'll get around to it is not yet clear, so I thought I'd let everyone know about the workaround. Remember, you can accomplish the same thing outside our toolset too: just create a new table, move the data, and swap names.

Why not Insanity?

2007-01-03T20:34:00Z

Writing the first entry in a blog has always seemed to be the hardest. Since this is a new year, and this is all about Something New, I thought I'd take the moment to blather about why I have made the change over from Dev to PM.

I'm part of the Visual Studio Team System for Database Professionals - affectionately known as "DataDude" - team. I signed up this summer because the team was looking for someone else (The team has Gert Drapers in it, so there was already plenty of SQL knowledge) with some Deep SQL background and some insight into real-world SQL applications to balance their/our development against. I came on board as a Dev under Gert, and had some small role to play in the push to get V1 out the door (which happened back in the beginning of December).

But I'm greedy. I wanted a broader scope, with more opportunities for Customer contact and keeping my finger on the pulse of what customers - and potential customers - might expect from our product. It's one of the reasons I came on board in the first place, and I was getting a bit concerned that if I sat behind a desk and did nothing but code, I might lose that perspective. So one of the perks of being a PM is that getting out in front of the customers is Part Of The Job.

And, admittedly, I love the conference circuit. Back when I was in MSR (that's Microsoft Research, for those who might not know), I was the PM for the Community Technologies Group, and I got to go to TechEd's and PDC's and evangelize the work we were doing in the Newsgroup space (I'll have to blog more on that some day). Anyway, I thoroughly enjoy that process, and it was something that I really wanted to do again. And face it, there's less opportunity to do that as a developer than as a PM. Now I'm already looking forward to being in SF for the VSLive conference this coming March, and there's more in queue.

At the same time, I don't have to give up Dev entirely - a lot of sample code and proof of concept work is done by PMs. I've spent the last year and a half doing C# (as a 10+ year VB dev, it wasn't that painful at all!) and wouldn't want to let any of that rust! And as we look forward, understanding how potentially the DataDude project might impact the App Tier code is going to be part of the investigation, so...

Anyway, here I am - new year, new job, new challenges. It's all very exciting - with a dash of cautious apprehension. Keep checking in on me - and I'll try to keep something entertaining going on here. Later!