Being Cellfish : SQL

How to prevent the use of "SELECT *"

cellfish — Sun, 08 Nov 2009 16:11:00 GMT

One good tip for writing good SQL code is to always select the columns you need and never just select everything. Well here is a description on how to actually prevent SELECT * queries (you need to complete a free registration to see the article). The basic idea is to add a dummy column to all your tables and then limit the access to this column using DENY SELECT ON OBJECT for that column. Personally I think this is maybe taking it a little bit too far but desperate times calls for desperate measures, right...

SQL programming mistake #12

cellfish — Mon, 28 Sep 2009 01:36:00 GMT

I would like to add another thing to my previous list. And that is the mistake of not properly understanding how to use correctly create the WHERE clause. I came to think about this when I read this which also includes a few good tips on how to rewrite your query to work more efficient.

SQL programming mistakes

cellfish — Mon, 14 Sep 2009 07:13:00 GMT

I recently read this comprehensive list with ten common mistakes done when writing SQL. And I'd like to add number eleven to that list: using dynamic queries in stored procedures. Especially in projects with a lot of stored procedures for parts (or all) of the business logic there is often a few stored procedures with which builds an SQL query in a string and then executes it. These tend to be very difficult to understand and frankly are often only there just in order to return exactly what is needed. Personally I prefer a more naive way to solve queries that just have to be dynamic. There are three things you can do depending on exactly what makes the query complex. Either you can create a single query that returns everything you possibly need and then filter the data afterward. Or you can get what you need in a number of smaller queries. The last alternative I tend to avoid is to move the creation of the SQL out of the stored procedure and keep it as parameterized as possible.

All in all, in my experience the use of dynamic queries is usually not needed at all. Only with a bad database design may they really be needed,in which case you need to work around it as I described above. But the best way to avoid it is to make sure the design of the database supports the kinds of queries you need to do instead of relying on dynamic queries.

20 tips to write a good stored procedure (is really just 12)

cellfish — Wed, 12 Aug 2009 18:12:00 GMT

A few days ago there was an article with 20 tips to write a good stored procedure (requires free registration to read). The problem is that there are really only 12 good tips (and 4 bad and 4 neither good or bad). So let me go over the tips one by one and comment on them:

Capital letters for keywords and proper indentation. With todays code editors with syntax high lighting I don't see why you want to high light keywords with capital letters. The code editor will do that for you. And suggesting proper indentation is not really a tip to write a good stored procedure. It's common (coding) sense! So I don't think this one counts... Score (good-not really-bad advice IMHO): 0-1-0.
Use SQL-92 syntax for joins. If MS SQL server drops support for the old syntax this is good advice. Score: 1-1-0
Use as few variables as possible. The article mentions cache utilization as an argument. Sounds like premature optimization to me. I'd say use as many variables as makes sense to make the code most readable. If that turns out to be a problem, then you optimize. So in general I found this advice to be bad. Score: 1-1-1
Minimize usage of dynamic queries. Kudos to the article to pointing out how to minimize the bad of dynamic queries and I guess technically minimizing could mean zero but that is really the only good advice; don't use dynamic queries. So once again a bad, or at least misleading advice IMHO. Score: 1-1-2
Use fully qualified names. If you don't do this you might end up with some weird behavior so this is a good advice. Score: 2-1-2
Set NOCOUNT on. Good advice: Score: 3-1-2
Don't use sp_ prefix. Score: 4-1-2
KEEPFIXED PLAN. Learn from this article and use it correctly. Hard to argue with "learn something and use it right". Score: 5-1-2
Use select instead of set. Once again performance is mentioned as a motivator. However the potential bad side effects of using select rather than set are more important in my opinion. The problem with select is that the variable might not be set if a query returns no rows and the set gives you an error if the select returns more than one row. Read more about it here. I'd definitely prefer set over select. Score: 5-1-3
Compare the right thing in the where clause. This advice just confuses me. The article talks about what operators are the fastest and then refers to this page talking about preference. Even though the article is confusing on this point the basic idea is correct. For example using IN is generally faster than NOT IN. So I'll call this one a draw. Score: 5-2-3
Avoid OR in WHERE clause. This is good advise for good performance. Score: 6-2-3
Use CAST over Convert. CAST is SQL92 standard. Convert is not. Score: 7-2-3
Avoid distinct and order by. Once again this is common sense. Don't do things you don't need... Score: 7-3-3
Avoid cursors. This falls into the same category as dynamic queries to me. The only good advice is don't use cursors. Score: 7-3-4
Select only the columns you need. Common sense! Score: 7-4-4
Sub queries vs joins. Article lists a few good rule of thumbs. I think you should use whatever is most readable. Score: 8-4-4
Create table vs select into. Article points out important differences. Score: 9-4-4
Use variables instead of temporary tables. Score: 10-4-4
Use proper indexes. Score: 11-4-4
Use profiler. Many tips in the article suggest you do things to improve performance. But I think doing so before you know you have a problem is a waste of time and resources. So this advice is actually one of the best advices in the article. Score: 12-4-4

Instead of triggers

cellfish — Thu, 09 Jul 2009 08:15:00 GMT

I've never been a big fan of triggers in a database since the existence of triggers is easily overlooked. I've also seen the use of triggers in situations where they where used to fix a bad database design. Specifically trigger were used to enforce constraints on other tables. I've also seen triggers used to move data to another table to keep history of records. I think that use is debatable since a stored procedure can be used to do that and the application should use the stored procedure and not do delete and updates directly. So in this case the trigger was used to protect the system from developers not following the application design. This should not really be needed but...

Anyway, since I've not been working with triggers for quite a while I've never really thought about the options available but today I saw this video (requires free registration) which covers instead of triggers. I think it's a nice feature which simplifies the implementation of protect-from-developers type triggers since you can basically redefine a delete to not do an delete at all and vice versa (if you have the urge to delete data instead of updating it).

But in my opinion triggers is not the best way to handle things in your application. Use a stored procedure since a trigger easily is forgotten but a used stored procedure is not. Also it is easier to follow the flow in a stored procedure over trigger execution. And if you have a nice stored procedure everybody should use but your developer keep screwing things up by using inserts, updates and deletes directly in the database then I would consider adding a trigger preventing direct use. This can be hard since the stored procedure it self will cause the trigger to execute but in my experience the stored procedure typically updates only one record at the time so an easy way to try to find developer "abuse" is to prevent updates of more than one row at the time. That has been enough for me so far.

Preemptive comment response: I know instead of triggers is nothing new and has been available for some time but I've just not been interested in triggers for the last 9 years so please forgive me...

Random SQL

cellfish — Wed, 10 Jun 2009 08:12:00 GMT

I must say that I have never used the random functionality in SQL more than once in my life and that was not for work, it was in a hobby project. Never the less, I find this article (free registration required) on how the RAND function works and is seeded quite interesting.

A summary of the article: You want a random number for each row in a result to do some kind of filtering. The problem is that the RAND function by default uses date/time to seed the random so in a query you don't really get any different random numbers. So you need a nifty way to seed the random for each row. The article proposes: RAND(Checksum(Newid()))

The use of Newid in the link above might however not be practical for performance reasons. If you want to try some other ways to produce random you can find more alternatives here.

SQL Unit Testing

cellfish — Tue, 05 May 2009 06:05:00 GMT

Thought it was time to combine my old favorite subject (SQL) with a new interest (TDD). So how do you test drive your SQL? I think the first answer is you don't. The database is typically something you want to fake anyway since setting up and accessing the database is to slow when you want rapid feedback. Nowadays when things like NHibernate are popular I think most people don't write much SQL anyway so there is no need to test drive the creation of SQL queries or stored procedures. But there was a time when many applications kept a lot of logic in the stored procedures and in those systems it might make some sense. Also we have the fact that the database needs to be created some way. Why not test drive the database schema?

So even though this might not be the hottest topic on the block, what alternatives do we have? First of all we have SQLUnit. This project does not seem to have been updated for several years. Guess that is because we don't put all application logic in stored procedures any more. But it is at least an attempt to create a unit test framework for SQL. But why use the same language for the tests as the implementation? Some people think this is OK but I think there is also a danger in doing so. Switching languages typically makes you do small stupid mistakes you don't do if you stick to one language. But sometimes it might still be worth it. For example I'd much rather use any .Net based unit test framework over SQLUnit and an example of how that looks can be found here.

Then there is of course the option of not using any framework at all. Creating your own framework may be the most efficient way of doing things and an example of database creation in a test driven manner without any fancy frameworks can be found here.

Becoming a good programmer takes time

cellfish — Tue, 31 Mar 2009 07:24:00 GMT

By now you've hopefully heard about craftsmanship over crap and the other day I heard about a research report where they concluded that university students majoring in engineering have problems translating theory into practice (article in Swedish). Education is not the key to becoming a good programmer. It is a catalyst that will help you get better faster. But it will always take time to become a good developer. But to become a great developer you must never stop learning. So why am I talking about this? Well, it's because something I read that made me real sad:

"Benefits: [...] Through less lines of code you can get more done."

And this is not the only example from the DBA world I've seen. First of all the statement is false. Through less lines of code you get the same work done. But less is not more. Just reducing the number of code lines for the sake of reducing lines generally result in code more difficult to understand and maintain. For some reason being a DBA and being a developer seams like a rare combination. I've worked with a few DBAs that worked like developers mainly because they were developers that had an interest in databases and then turned into DBAs. And what do I mean with "working as a developer"? Being lazy, i.e. automating as much as possible and making sure they can reuse stuff without duplicating code. Maybe even having a number of automated tests. And having at least basic understanding of what makes code easy to understand and easy to maintain.

Joining with previous and/or next row

cellfish — Fri, 23 Jan 2009 12:23:00 GMT

Sometimes in my career I've had to create SQL queries that uses data from the previous and/or next row in the result. That is, something I wont on row X depends on row X-1 or X+1 in the result. Most of the time I have not done this in the SQL query itself but rather in worked on the result but a few times I just had to do it in SQL. This leads to a construction with sub-selects so the resulting query is both hard to read and understand and it is slow due to a large number of sub-selects.

This morning I read a short article describing the use of row_number method introduced in SQL Server 2005. Worth reading if you need to solve similar problem since it removes the need for sub-selects. At least the same amount of sub-selects. To read the article you have to register with the site but it is free and you don't get spammed.

Testdriven Database development

cellfish — Wed, 29 Oct 2008 13:10:00 GMT

I recently found this interesting article on TDD when using T-SQL. It describes how a new unit test framework form T-SQL, called tsqlunit, works. Even though it feels like the framework lacks a lot of the fancy asserts and that is probably because of limitations in T-SQL I still welcome it as a new addition in the family of unit test frameworks. Definitely something I'll use next time I have to do T-SQL work.

Be suspicious to DAL frameworks

cellfish — Wed, 16 Jul 2008 23:10:00 GMT

I've always been suspicious to SQL queries that are automagically generated by some framework. And when I read this article on lengthy SQL queries it certainly was another gallon of gasoline on the fire. Sure premature optimization is the root of all evil and all that but there is also another important rule in software development; Don't do obviously stupid things. If you want to use a framework for data access, which is very common for productivity reasons, be sure to design your software so it is easy to replace the generic framework with something specific.

If you on the other hand end up with really large queries when you write the queries your self (I'm a stored procedure guy so I have a hard time even making up what kind of SQL query would end up being that large), but it is obvious what the solution is - Stored procedure.

Readable SQL

cellfish — Thu, 12 Jun 2008 09:25:00 GMT

I read this article on SQL Code Layout and Beautification and can only agree with the author that other people's SQL often is hard to understand because I'm so used to how I write my SQL statements. One of the links is to an on-line tool formatting your SQL according to a number of rules. I was happy to find that my way of writing SQL was supported:

SELECT a,
       b,
       c AS d
FROM x,
       y,
       z AS w
WHERE a = 2
   AND b IN (3,4,7)

And another example:

SELECT obj.run,
         obj.camcol,
         STR(obj.field,3) AS field,
         STR(obj.rowc,6,1) AS rowc,
         STR(obj.colc,6,1) AS colc,
         STR(dbo.FOBJ(obj.objid),4) AS id,
         STR(obj.psfmag_g - 0 * obj.extinction_g,6,3) AS g,
         STR(obj.psfmag_r - 0 * obj.extinction_r,6,3) AS r,
         STR(obj.psfmag_i - 0 * obj.extinction_i,6,3) AS i,
         STR(obj.psfmag_z - 0 * obj.extinction_z,6,3) AS z,
         STR(60 * distance,3,1) AS d,
         dbo.FFIELD(neighborobjid) AS nfield,
         STR(dbo.FOBJ(neighborobjid),4) AS nid,
         'new' AS 'new'
    FROM (SELECT obj.objid,
                 run,
                 camcol,
                 field,
                 rowc,
                 colc,
                 psfmag_u,
                 extinction_u,
                 psfmag_g,
                 extinction_g,
                 psfmag_r,
                 extinction_r,
                 psfmag_i,
                 extinction_i,
                 psfmag_z,
                 extinction_z,
                 nn.neighborobjid,
                 nn.distance
            FROM photoobj AS obj
                 JOIN neighbors AS nn
                   ON obj.objid = nn.objid
           WHERE 60 * nn.distance BETWEEN 0 AND 15
             AND nn.mode = 1
             AND nn.neighbormode = 1
             AND run = 756
             AND camcol = 5
             AND obj.TYPE = 6
             AND (obj.flags & 0x40006) = 0
             AND nchild = 0
             AND obj.psfmag_i < 20
             AND (g - r BETWEEN 0.3 AND 1.1
                  AND r - i BETWEEN -0.1 AND 0.6)) AS obj
         JOIN photoobj AS nobj
           ON nobj.objid = obj.neighborobjid
   WHERE nobj.run = obj.run
     AND (ABS(obj.psfmag_g - nobj.psfmag_g) < 0.5
           OR ABS(obj.psfmag_r - nobj.psfmag_r) < 0.5
           OR ABS(obj.psfmag_i - nobj.psfmag_i) < 0.5)
ORDER BY obj.run,
         obj.camcol,
         obj.field

SQL crosswords

cellfish — Thu, 29 May 2008 11:29:00 GMT

I don't know what's worse; making crosswords with only SQL related questions (requires free registration to access) or actually trying to solve the same. This is one of many thing that currently puzzles me. One other thing that also scares me is that I get annoyed when I cannot solve these crosswords right away...

Stored Procedures and data sharing

cellfish — Thu, 15 May 2008 15:45:00 GMT

In my career I've worked with stored procedures in a way I think many people never do. For example I've worked on projects where both the data access and the business logic have been placed in the database in the form of stored procedures. Not quite your standard three-tier solution but it works quite well when the people with all the business knowledge also knows a lot about databases and SQL but less about anything else. Obviously this involves a lot of data being passed around between stored procedures.

This means that I've seen a few different ways to pass data around and I'm sad I didn't find this page earlier. Sommerskog's description of the pros and cons with different approaches would definitely have saved be some time. Over the years I've tried output parameters, cursors, tables, user defined functions and the table data type. And as usual the latest technology is not the best for all situations but may simplify things that was cumbersome to accomplish before.

And some times the simplest solution is the most brilliant. A common problem I've faced is that I have one SP (stored procedure) returning some data as a select statement. Later on I want to reuse that SP in several other SPs. Previously I've been using functions or temporary tables to do this which both have their disadvantages, rewriting the original SP being one. But with the insert-exec pattern I can reuse the SP right away. I'm embarrassed I didn't think of that before. Let's say it is so obvious you don't think about it. In the same way you forget to look for your glasses on your nose. It's no silver bullet and it wouldn't have solved all my problems but it is yet another tool to use.

So if you're working with databases you should definitely take a closer look at what Sommerskog is writing because knowing things like that will definitely make your life easier. And there is no point in learning it the hard way by your self.

UPSERT in SQL server 2008

cellfish — Tue, 29 Apr 2008 09:04:00 GMT

I recently read this article on upsert functionality in SQL server 2008. I thought that the SQL server team finally had come to theri senses and added functionality similar to replace in MySQL, something that have been available in MySQL for ages. But no. The article describes a way to use the new merge command in order to perform an "upsert".

An upsert is an operation where rows in the database are updated if they exists and inserted if they don't. This has been available as a replace statement in MySQL for quite some time. At first glance the merge seems much more cumbersome to use than the replace, but don't be fooled by this. One problem I've had with the replace command is knowing exactly what rows will be updated and which will be added. Especially when updating things that are part of a multi-column primary key. The merge however is really more clear in what it actually does and you have full control of what you update and what you insert.