Rico Mariani's Performance Tidbits : databases

Linq Compiled Queries Q & A

ricom — Mon, 25 Aug 2008 22:21:00 GMT

I did a series of postings on Linq Compiled Queries last year, I recently got some questions on those postings that I thought would be of general interest.

Q1:

Why use the 'new' keyword in this snippet?

var q = from o in nw.Orders
select new {o.everything …};

If you did just :

var q = from o in nw.Orders
select o;

You're getting editable orders. Linq then has to track them in case you change them and want to submit the changes. If you use new effectively you're making a copy of the orders that is not going to be change tracked. That's faster for read only cases. The other thing you can do is mark the query context as read-only and then you get the same effect. When I wrote that test case, that feature wasn't available yet so I used new to simulate it.

Q2:

What do you mean when you say that linq will 'Create custom methods that bind the data perfectly' ?

Whenever you use linq to sql to read data from a database it has to do two important things for you. The first is convert your Linq query into SQL. The second is to make a method that takes the stream of data that comes back from the database and converts it into the managed objects you required. That's the data-binding step. Linq creates the necessary methods automatically, and it makes the perfect code for doing this.

Q3:

How did Linq to SQL beat your ADO.Net code for insert times. Shouldn't a tie be the best possible result?

The SQL I used in my test case was pretty much the standard simplest SQL you would use for such a job. The automatically generated SQL from Linq was better than what I wrote by hand because they had parameterized the insert statements which I never bothered to do. Had I changed my SQL to what they created it would have been a tie. This is kind of like when the C++ compiler finds a machine code pattern that is better than what you would have written doing it by hand because it did something you don't usually bother doing with hand tuned machine code. But you *could* replace what you wrote with what the compiler generated.

Q4:

What are the downsides to precompiled queries?

There is no penalty to precompiling (see Quiz #13). The only way you might lose performance is if you precompile a zillion queries and then hardly use them at all -- you'd be wasting a lot of memory for no good reason.

But measure :)

Performance Quiz #13 -- Linq to SQL compiled query cost -- solution

ricom — Mon, 14 Jan 2008 20:51:00 GMT

Well is there really a "solution" at all in general? This particular case I think I constrained enough that you can claim an answer but does it generalize? Let's look at what I got first, the raw results are pretty easy to understand.

The experiment I conducted was to run a fixed number of queries (5000 in this case) but to break them up so that the compiled query was reused a decreasing amount. The first run is the "best" 1 batch of 5000 selects all using the compiled query. Then 2 batches of 2500, and so on down to 5000 batches of 1. As a control I also run the uncompiled case at each step expecting of course that it makes no difference. Note the output indicates we selected a total of 25000 rows of data -- that is 5 per select as expected. Here are the raw results:

Testing 1 batches of 5000 selects
5000 selects uncompiled 9200.0ms 25000 records total 543.48 selects/sec
5000 selects compiled 5401.0ms 25000 records total 925.75 selects/sec

Testing 2 batches of 2500 selects
5000 selects uncompiled 9181.0ms 25000 records total 544.60 selects/sec
5000 selects compiled 5402.0ms 25000 records total 925.58 selects/sec

Testing 5 batches of 1000 selects
5000 selects uncompiled 9169.0ms 25000 records total 545.32 selects/sec
5000 selects compiled 5432.0ms 25000 records total 920.47 selects/sec

Testing 100 batches of 50 selects
5000 selects uncompiled 9184.0ms 25000 records total 544.43 selects/sec
5000 selects compiled 5511.0ms 25000 records total 907.28 selects/sec

Testing 1000 batches of 5 selects
5000 selects uncompiled 9166.0ms 25000 records total 545.49 selects/sec
5000 selects compiled 6526.0ms 25000 records total 766.17 selects/sec

Testing 2500 batches of 2 selects
5000 selects uncompiled 9165.0ms 25000 records total 545.55 selects/sec
5000 selects compiled 7892.0ms 25000 records total 633.55 selects/sec

Testing 5000 batches of 1 selects
5000 selects uncompiled 9157.0ms 25000 records total 546.03 selects/sec
5000 selects compiled 10825.0ms 25000 records total 461.89 selects/sec

And there you have it. Even at 2 uses the compiled query still wins but at 1 use it loses. In fact, the magic number for this particular query is about 1.5 average uses to break even. But why? And how might it change?

Well, as has been observed in the comments, Linq query compilation isn't like regular expression compilation. In fact compiling the query doesn't do anything that isn't going to have to happen anyway. In fact, actually creating the compiled query with Query.Compile hardly does anything at all, it's all deferred until the query is run just as it would have been had the query not been compiled. So what is the overhead? Why is it slower at all? And what's the point of it?

Well the main purpose of that compiled query object is to have an object, of the correct type, that also has the correct lifetime. The compiled query can live across DataContexts, in fact it could potentially live for the entire life of your program. And since it has no shared state in it, it's thread-safe and so forth. It exists to:

1) Give the Linq to SQL system a place to store the results of analyzing the query (i.e. the actual SQL plus the delegate that will be used to extract data from the result set)

2) Allow the user to specify the "variable parts" of the query. The most common case isn't that the query is exactly the same from run to run, usually it's "nearly" the same... That is it's the same except that perhaps the search string is different in the where clause, or the ID being fetched is different. The shape is the same. Creating a delegate with parameters allows you to specify which things are fixed and which things are variable.

Now there was some debate about how to make compiled queries durable, automatically caching them was considered, but this was something I was strongly against. Largely because of the object lifetime issues it would cause. First, you would have to do complicated matching of a created query against something that was already in the cache -- something I'd like to avoid. Secondly you have to decide where to store the cache, if you associate it with the DataContext then you get much less query re-use because you only get a benefit if you run the same query twice in the same data context. To get the most benefit you want to be able to re-use the query across DataContexts. But then, do you make the cache global? If you do you have threading issues accessing it, and you have the terrible problem that you don't know when is a good time to discard items from the cache. Ultimately this was my strongest point, at the Linq data level we do not know enough about the query patterns to choose a good caching policy, and, as I've written many times before, when it comes to caching good policy is crucial. In fact, analogously, we had to make changes in the regular expression caching system back in Whidbey precisely because we were seeing cases where our caching assumptions were resulting in catastrophically bad performance (Mid Life Crisis due to retained compiled regular expressions in our cache) -- I didn't want to make that mistake again.

So that's roughly how we end up at our final design. Any Linq to SQL user can choose how much or how little caching is done. They control the lifetime, they can choose an easy mechanism (e.g. stuff it in a static variable forever) or a complicated recycling method depending on their needs. Usually the simple choice is adequate. And they can easily choose which queries to compile and which to just run in the usual manner.

Let's get back to the overhead of compiled queries. Besides the one-time cost of creating the delegate there is also an little extra delegate indirection on each run of the query plus the more complicated thing we have to do: since the compiled query can span DataContexts we have to make sure that the DataContext we are being given in any particular execution of a compiled query is compatible with the DataContext that was provided when the query was compiled the first time.

Other than that the code path is basically the same, which means you come out ahead pretty quickly. This test case was, as usual, designed to magnify the typical overheads so we can observe them. The result set is a small number of rows, it is always the same rows, the database is local, and the query itself is a simple one. All the usual costs of doing a query have been minimized. In the wild you would expect the query to be more complicated, the database to be remote, the actual data returned to be larger and not always the same data. This of course both reduces the benefit of compilation in the first place but also, as a consolation prize, reduces the marginal overhead.

In short, if you expect to reuse the query at all, there is no performance related reason not to compile it.

Performance Quiz #13 -- Linq to SQL compiled queries cost

ricom — Fri, 11 Jan 2008 23:08:00 GMT

I've written a few articles about Linq now and you know I was a big fan of compiled queries in Linq but what do they cost? Or more specifically, how many times to you have to use a compiled query in order for the cost of compilation to pay for itself? With regular expressions for instance it's usually a mistake to compile a regular expression if you only intend to match it against a fairly small amount of text.

Lets do a specific experiment to get an idea. Using the ubiquitous Northwinds database and getting the same data over and over to control for the the cost of the database accesses (and magnify any Linq overheads) we run this query:

var q = (from o in nw.Orders
            select new {
                OrderID = o.OrderID,
                CustomerID = o.CustomerID,
                EmployeeID = o.EmployeeID,
                ShippedDate = o.ShippedDate
           }).Take(5);

and compare it against:

var fq = CompiledQuery.Compile
(
    (Northwinds nw) =>
            (from o in nw.Orders
            select new
                   {
                       OrderID = o.OrderID,
                       CustomerID = o.CustomerID,
                       EmployeeID = o.EmployeeID,
                       ShippedDate = o.ShippedDate
                   }).Take(5)
);

So now the quiz: How many times to I have to use the compiled version of the query in order for it to be cheaper to compile than it would have been to just use the original query directly?

Database Performance, Correctness, Compostion, Compromise, and Linq too

ricom — Sat, 01 Sep 2007 04:10:00 GMT

Introduction and Disclaimer

Regular readers of my blog are already familiar with my goal to provide brief and useful information that is approximately correct and that illustrates some key truths. Most of the time my articles are not authoritative and that is especially true in this case. I am certainly not an Official Microsoft Authority on databases and data systems, I just have a good bit of experience in that area, and I wanted to convey some things I learned that I thought were important, and that I’ve never seen assembled as a whole before, so I’ve written this article. This article uses Linq to SQL for its examples but I think it is actually more broadly applicable, with due caution.

Performance in Many Tier Systems

Again if you read my blog you’ll know that I always talk about the importance of measuring performance in context. This is especially important in systems with multiple tiers because making a choice in one tier can profoundly impact the consequences in other tiers. For instance, a client-side cache might take a lot of load off of the middle tier. Sounds great right? Oops, you of course remembered that if you cached the contents on the client then they are going to be at least a little of out of date right? You’re ok with that? Or did you add a scheme to periodically recheck validity. Oops, now you have more traffic to the middle teir again. No, wait, you have a nice periodic discard policy where you only keep local data for a known interval? Ah yes, but now your cache contents aren’t necessarily self-consistent because they were fetched with different queries at different times and unified to create the cache. Does it ever end?

There is a dance here, and it is a complicated one. Only by understanding how all the dancers play together in your system can you truly create systems that have solid correctness characteristics and good performance. And it’s a bad idea to look at the performance of any one of the dancers independently.

Key Factors

I break it down into a fairly small list of considerations, and these are of course deeply entangled.

Locality
Isolation
Unit of Work

You might wonder why I’m not mentioning things like network, schema, and so forth… It’s because I think about them in the context of the bigger phenomena. And maybe you’re wondering what this could possibly have to do with Linq to SQL but don’t worry we’re going to get there by the time we talk about unit of work.

Locality

Even when we’re not talking about databases I often say “Locality is everything” and it’s possible that it is even more true in the database world (don’t nitpick me on “degrees of truth”, it’s only a figure of speech ^_^). When you’re talking about a data system, bad locality generally translates directly to more disk operations which in turn translate into torpedoed performance.

What do we do about it? Two big things: Schema and Indexes. Maybe that’s really two sides of the same thing but let me separate those concerns for a moment.

We create schema in a manner that is consistent with the data we intend to represent and so that logically related concerns will tend to be physically together. If we normalize well we also tend to find that a single logical fact tends to be represented in a single physical location. All of this bodes well for locality.

But now we’re faced with a problem. One organization of the data is frequently not enough to support all the operations that we intend to perform on the data. If we want to look up only by (e.g.) ID, one ordering is fine but the moment that we want to look up also by (e.g.) surname then we need to organize the data in more than one way. That is where indexes come in to play.

Trust me, I’m going somewhere with all of this.

To get multiple orderings of the same data you could create multiple tables with the same data arranged differently. That would work but it would be highly inconvenient as you would always have to be choosing the flavor of the data to access when you queried the data and you would have to update multiple copies of the data whenever you made changes.

Indexes let you automate this. Indexes are your way of telling a database that you want multiple copies of your data, ordered differently, and any time you do an operation that updates the main copy you want all of the alternate copies also updated automatically in the same transaction.

This is actually a profound statement. Remember we added indexes because there were some questions we could not answer without looking at a lot of data (bad locality) and the price we pay is that now when we make changes to one spot in the data we actually have to propagate that change to many places atomically (reduced locality).

In the final analysis an index isn’t much more than a second, or third, etc. copy of the table with some of the columns reordered and some removed entirely.

Locality means we get just the data we need, that it’s organized in such a way that we do not have to sift though vast volumes of data to get to what we want, and that we can make surgical changes to the data in support of the things our system needs to do as it runs. Good locality translates directly into good performance.

Isolation

Isolation is a notion that is cloaked in mystery, perhaps unnecessarily. I’ve met many developers that know what a transaction is but far fewer that know about isolation levels and fewer still that understand how these things are entangled. In short, isolation is about giving every program using a data system the illusion that it is the only user. Levels of isolation, roughly, describe how imperfect that illusion is going to be.

OK if you’re still with me at this point you must be wondering what any of this could possibly have to do with performance and even more wondering why on earth I am now talking about Isolation, a concept that is understood even less well than Locality.

I’m glad you asked :)

The first thing you should know is that the better your data locality the easier it is to create the illusion of isolation. The simplest case is two clients looking at two completely unrelated sections of a database – they have no overlap in their operations whatsoever and so isolation is easy. Now of course the more data you access the greater the likelihood that you will overlap with someone else and some isolation technique is going to be necessary to preserve the illusion. If your data has great locality then you’ll be able to make nice tight queries minimizing your isolation needs.

The second thing you should know is that maintaining isolation has a cost, like everything else. Depending on the technique used it can have both direct costs and indirect costs. But that’s really abstract so let’s be more concrete to illustrate these costs with a specific example.

Let’s suppose I have a simple database with just one table in it. It’s a set of account numbers, names, and balances. Now just one table isn’t enough to really model a bank with necessary auditing and so forth but this example is already complicated enough to show some isolation concerns and how they turn into performance problems. Furthermore let’s say the isolation required is one of the most basic choices “READ_COMMITTED” meaning I’m never allowed to observe someone else’s data if they have not yet committed their transaction – so no intermediate results.

Now let’s suppose I’m a big customer of this bank and I have 100 accounts, conveniently numbered 1000, to 1099. Some of the accounts are in my name and some of them are in my wife’s name. I want to know the total amount of money in the accounts in my name. In SQL it would look like this

select sum(balance)
from account
where account.id > = 1000 and account.id <= 1099 and account.name = ‘Rico’

The situation doesn’t get much easier than that. We could assume there is a nice index on the account table by ID so that all those accounts are (nearly) contiguous and with one scan of just my slice of the data we can get the answer. That’s about the best locality we could hope for.

Now you can imagine that you are the little database engine, you go and read account #1000, check the name, find that it’s Rico, and add the balance. Move on to #1001, check the name, add the balance. Chug chug chug.

Now let me complicate things just a little bit. While this is happening, and we’re on account #1050 (chug chug) another user comes along and deposits money into one of my accounts. Let’s say its account #1060. Is this a problem? Well no it isn’t. Since I haven’t yet read the contents of account #1060 when I get there I will find the new balance and all is well. Or is it?

Here’s a subtle point: the total that I get isn’t guaranteed to be the total that I would have gotten if the whole query had run at the instant that that it started (you could imagine such a system but that’s unusual) the only guarantee is that we will get some total that represents the sum of the balances as they could have existed at some moment where all the data was committed. So here we’ve created valid total, an especially interesting one because it shows the balance at the instant the query finishes.

Great, so we can write into areas we have not yet read without any trouble.

In fact we could even do something a little more complicated. Suppose we did a transfer of $100 from one of my accounts, #1060, to another account, #1070. We’re still in good shape because neither of those two accounts has yet been read and so the sum will be computed correctly.

Hey, this isolation thing doesn’t seem very hard so far! I wanted to subtract money from #1060 and put in #1070 and everything is great. But what if I had wanted to put the money in account #1030?

Now I have a problem, #1030 has already been read. If I allow the money to be moved then the sum calculation will be off by $100 because the $100 is gone from #1060 which I have not yet read and it was missing from #1030 which I have read. Ooops.

What do we do about this? Well there are many approaches, I will pick one for this example, the key thing to remember is that all the approaches have costs.

You might think that what the database must do when the query begins is lock everything that is going to be read and prevent anyone/everyone from writing to those things. That could work, if only it could predict, accurately, what it is that will be read. Or it could lock more than will be read, hopefully not much more, that could work too. But keep in mind these things:

#1 predicting the future is hard (e.g. in this example, which rows belong to me and which belong to my wife?)

#2 we want to lock as little as possible so that as much can continue running as possible (e.g. all updates to my wife’s accounts would be fine since they do not affect the sum)

So we might end up at a different scheme – rather than guess the future, lets lock the past. When the transaction wrote to #1060, subtracting $100 that was just fine, so far. The contents of #1060 are still dirty so if the reader arrives there it must wait for the transaction to finish. Meanwhile, if the writer chooses to move the $100 to account #1070 all is well. #1070 has not been read yet, it isn’t locked, the writer moves the cash, commits the transaction and if the reader had been waiting for #1060 to finalize it will do so and the read operation can proceed. So perhaps the summing was paused for a moment but otherwise everything went well.

Notice that we already have taken a bunch of performance hits. We had to mark rows that were “dirty” and pending a transaction with some kind of write lock. We had to have readers checking the write locks and waiting (i.e. taking longer) if they encountered a lock. And we had to clean out the write locks whenever a transaction committed. But that was the easy case.

What if, after subtracting $100 from account #1060 we then attempted to add the $100 to account #1030. Lucky for us new “lock the past” isolation scheme says that row #1030, since it has already been read, now has a read-lock. That means we can’t write to it. But wait. #1060 has a write lock. So when the reader eventually gets to #1060 it will stop. It will not give up its read lock on #1030 so the write operation cannot complete, meanwhile the read operation on #1060 cannot complete and so the read long on #1030 well never be released.

Deadlock.

You’ll notice I’ve managed to create a deadlock with just one writer in the system merely because of the need for isolation.

Now you can imagine other isolation methods that would not get a deadlock in this case but they have other costs and we’re not studying them. The point here is that there is a direct isolation cost, in any scheme, and now we’re about to see an example of an indirect cost.

Since the system is deadlocked, the database must now choose one of those two transactions (the reader or the writer) and abort it.

If it chooses the reader, then the write will be allowed to complete, the reader must retry the operation and then get the correct and accurate new sum based on the completed write transaction.

If it chooses the writer, then the reader may complete but first we must use the transaction log to restore the original contents account #1060 (i.e. the transaction does not commit, it aborts) and we get the correct sum. Meanwhile the writer must retry its operation.

Can you see indirect costs? Probably the most significant is that either the read or the write must be retried. Now these were both tiny cases but you can imagine if many accounts were implicated and if there was complicated math, sorting, etc. that redoing that read query could be costly and likewise the write query with many indices and if it spanned many tables could be rather complex, to say nothing of the business logic that might be required to redo the math. In this case it’s a simple add $100, subtract $100 but in a complex system it might be a tricky reservation computation, mileage credit and commission distribution – with auditing. Anything you have to redo is just wasted work.

Importantly, the above is a normal and natural part of using a database and nothing has gone “wrong” here and we can reach this vital conclusion. The more we read or write in one unit, the greater the amortized cost of isolation. The more we attempt to write in one unit the greater the chance that the entire write will have to be aborted (and then redone from scratch).

This leads us right into the next topic…

Unit of Work

More than anything else how much work you do in one chunk drives everything else. Sure you need good locality – via a proper schema and indexes – and yes you need to choose the right isolation technique, where you have options, but you can destroy a system by trying to do too much at once – and people do.

If you’ve been waiting for me to make the connection to Linq here it is: every data layer has to make choices about how much to read and when. It may seem like a good idea to do large amounts of pre-reading, and in fact when measured in isolation it may seem like you get better performance when doing so, but, like the prisoners’ dilemma, when composed with the other operations that are happening on the server you may find that making the best looking choice independently results in a poor situation for the system as a whole.

Should you make the choice that is best for one part of the system at the expense of the system as a whole? Probably not.

Linq to SQL is faced with very real choices around how much to read in one chunk and how much isolation to guarantee. Creating a solution where we appear to offer excellent but non-composable performance would simply be leading people down a path to disaster – and you know I always advocate the Pit of Success. The obvious way should work out well.

Correctness

Once you’ve thought about unit of work you soon realize that you cannot afford to submit large transactions to your database – they are too likely to not commit successfully. So what do you do?

A classic thing to do is to break up those transactions into smaller pieces, but doing so creates a new set of intermediate stored results which must be valid. It’s a designed “workflow” for your uber-transaction where for instance every-line item in an order, or every 10 or something like that, is independently written and the order is flagged as “in flight.” In that world “in flight” orders are a normal healthy thing and your system has to be able to handle them appropriately – basically some intermediate states have become visible.

It sounds bad, but it’s essentially inescapable. The alternative is to *try* to commit gianormous transactions that really have no hope of *actually* committing with any kind of realiability. You may think that you’re going to get great performance by submitting all that work in one nice big chunk but you may find that all those savings are lost to the cost of all the extra retries you have to do to actually succeed.

And it gets worse.

If you have a client side component, like you do with Linq, then the data you have saved on the client might become “wrong” if something happens on the server that you don’t know about. You’ll need some kind of locking mechanism like Optimistic (what Linq uses) or Pessimistic. What this says is that before you write data back to the database you first re-verify that the data now in the database is still what it was before you made the update – if anything has changed out from under you then you throw an exception.

What does that mean? Well if you are writing a large amount of data and it has to be atomic that’s a bad thing because even if there wasn’t a database deadlock you still might find that some part of what you wrote has been altered. If you require that it all be as-it-was you may find you can never write anything, or that you often have to try 2, 3, 4, 5, 10, 50… who knows how many times to actually get the stuff to write.

What do you do about this?

We’re right back to the unit of work discussion. If you break those writes down into smaller chunks and make it so that you can write back in pieces – including some markers to show what is in flight and what is not – then you can “partly succeed” in your writes, even “mostly succeed” and when things go wrong because of a conflict you only need resolve those few records that actually did conflict and write them back to the database in your retry operation.

It is not wise to expect to successfully write thousands and thousands of rows in one operation and actually succeed on a production database under load.

Yes breaking those operations down will result in more round trips to the server and it may seem that such a thing performs more poorly (it will if measured alone) but those operations are much more composable with other things the server is going to be doing.

Your overall performance could be, almost certainly will be, A Lot Better (TM).

You’ll of course have to measure, in context.

One Last Warning

If you consider what I said, about the natural occurrence of failures in a database, then you’ll soon realize that it is *normal*, using Linq parlance, for db.SubmitChanges() to throw an exception from time to time. If you are trying to write a robust application with high reliability you need to think about that.

In addition to obvious things like, “the network went down”, “the database went down”, there are less obvious things like, “there was a deadlock”, “there was an optimistic lock conflict” that can and do happen. Those latter two things should be appropriately retried because *nothing bad has happened*. The strategy you choose, especially for cases where the optimistic lock failed, can have a profound impact on your performance and certainly you can’t just let those exceptions flow to the user. I think I can safely say that my mom doesn’t want to hear about how table X on connection A deadlocked with table Y on connection B.

If you’ve been reading carefully then you’ll see that it’s also “normal” for a foreach operation over a Linq query to fail from time to time – you need a retry strategy for those too to be fully robust.

Don’t get down on Linq though, those problems exist with all data solutions, the productivity benefits you get from Linq will go a long way to helping you to add the robustness you need in the areas you need it.

Don’t read “too much” at once. Don’t write “too much” at once. Handle deadlocks, they’re normal. Handle optimistic lock failures, they’re also normal. You should land in the Pit of Success.

DLinq (Linq to SQL) Performance (Part 5)

ricom — Mon, 16 Jul 2007 21:17:00 GMT

This posting is the last of what I had planned in this series but I think there are likely to be questions, especially when Orcas Beta 2 is more widely available so we're likely to talk about this some more.

First let's talk about the result I got and what it means. On my particular hardware I was able to achieve 93% of the throughput of the underlying provider.

How did I do this?

[7/17: Note: I just realized this could be read not how I intended. I mean 'how I did this' from the perspective of how the benchmark is built and how is it that the benchmark could be expected to achieve such a result. I did not write *any* of the Linq code myself, I only gave them some ideas to help improve this result. Matt didn't ask me to clarify this but he would have been more than justified if he had. :) ]

Well think about it, even though my test case is designed to magnify Linq overheads (because there is no business logic to do anything with the data and the data is all local and in the cache) what we've done by compiling the query is basically to remove most of the Linq overhead entirely. It's almost cheating except for you can do it too, and in meaningful, cases. In fact, arguably in the most common and important cases where you most need it you'll be able to get the best performance.

What are the major steps in running a query?

The expression tree that represents the query has to be created (var q = from etc...)
- You could avoid this even if you don't compile the query by just saving your query somewhere reusable
- If you weren't using Linq you'd have to just execute a command with parameters
The expression tree has to be converted to SQL
- The query can have parameters which in turn appear in the generated query
- The actual parameters are applied when the query is executed
- We make this a one-time cost by compiling the query once and saving the resulting SQL
The query has to execute
- The cost of this step is the same with Linq and without since the query looks very similar
- SQL server can use the same plan for the query since it looks the same each time
- This cost has been minimized in this test case by running the same query and keeping the database local
The results of the query have to be turned into objects
- In the May CTP this was done with reflection, however, now this is done by creating a custom method with light-weight code generation that does the object creation and data copying exactly the way you would do it by hand
- Since the columns that come back from the query are the same regardless of values of any parameters in the query this custom method can be re-used
- Instead of paying a cost to create the method every time you pay it once
- You still have to pay the cost of invoking this method via a delegate on each row, if you do it manually that code is inline so there is no function
Whatever you do with your data
- In this example it's basically nothing (one add operation)

So what's left?

We made #1, #2 and #4 one-time-costs. #3, and #5 we have to do in both cases no matter what. So what's the overhead? Not a whole lot, some checks to make sure we really can use the saved versions of everything and then the cost of calling the delegate. The reason the cost is as high as 7% is because so little is happening in steps 3 and 5. In real cases those steps would tend to be the bulk of your cost.

When can you get this benefit? Any time you are running the same (parameterized) query -- which is very often. Just as prepared statements and stored procedures are common/popular in regular SQL compiled queries should be popular in Linq to SQL. The most critical queries of your application can probably be compiled. Others might not be worth the hassle.

What about those insert and update cases?

In the Linq world, the update looks like a select, some data changes, followed by an update. I used the very same select statement and I arbitrarily updated the first dozen or so rows with a trivial update (I added 1s to a date field).

            var fq = CompiledQuery.Compile
                (
                    (Northwinds nw) =>
                            from o in nw.Orders
                            select o
                );

            int i = 0;
            for (; i < updateruns; i++)
            {
                using (Northwinds dc = new Northwinds(conn))
                {
                    int j = 0;

                    foreach (Orders o in fq(dc))
                    {
                        if (j++ > updatebatch)
                            continue;

                        o.OrderDate = o.OrderDate.Value.AddSeconds(1);
                    }

                    dc.SubmitChanges();
                }
            }

Note that I reported times for both the compiled case and the non-compiled case. That's because you can either compile the select part of the update or not. Depending on the frequency of actual updates you might find it worthwhile or not. Again this is a very dumb example designed to magnify Linq overheads.

Why did I get such outstanding performance?

The alternative code looked like this:

    
    // also runs in a loop updateruns times, not show here

    StringBuilder sb = new StringBuilder();

    while (dr.Read())
    {
        OrderDetail o = new OrderDetail();

        ... populate the fields...

        if (j++ > updatebatch) // updatebatch size was 10 in my test case
            continue;

        o.ShippedDate = o.ShippedDate.Value.AddSeconds(1);

        sb.AppendFormat("update Orders set ShippedDate = '{0}' where OrderID = {1}\r\n", 
               o.ShippedDate.ToString(), 
               o.OrderID);
    }






    // execute the query in the stringbuilder

That is all pretty simple stuff.. it's actually cheating a little because Linq to SQL will make sure that the data hasn't changed since it was read and I don't bother with that. However all of this is trumped by the fact that I didn't bother using prepared statements (but I did executed my updates in one batch) and Linq to SQL automatically made a prepared statement for doing the update and as a result SQL was able to process it better.

Could you do that yourself?

Yes. Would you? Maybe. Or you might use a stored proc to do the update for you. At that point my guess is that you would break even as you'd be back to doing exactly what Linq to SQL does. Isn't it strange that we're talking about what you have to do to the no-Linq case in order to get the speed you get with Linq by default? I think that's a good sign. But see the overall conclusions at the end.

What about the Insert case?

The insert test case gets its performance boost basically the same way except it's an batch of insert statements rather than updates and of course there is no select.

    using (Northwinds dc = new Northwinds(conn))
    {
        for (int j = 0; j < insertbatch; j++) // 10 items
        {
            Categories cat = new Categories();
            cat.CategoryName = "dummy_category" + j.ToString();
            cat.Description = "Description... Description... ' Description...Description...Description...Description...";

            dc.Categories.Add(cat);
        }

        dc.SubmitChanges();
    }

The batch insertion code in my test case (without Linq) looks nearly identical to the update case.

        sb.AppendFormat("insert into Categories (CategoryName, Description) values('{0}', '{1}')\r\n",
            "dummy_category" + j.ToString(),
            "Description... Description... Description...Description...Description...Description...");

So you can expect the Linq version will perform better for the same (artificial) reason as in the update.

What's my final word?

On selects you can pretty much make the Linq overhead vanish if you have a repeatable query pattern, which is a pretty common thing. That's great news.

On inserts and updates, my test cases weren't especially great and the main thing they illustrate is that good connection management and prepared statements dwarf the other costs in simple insert/update cases. The good news there is that Linq gives you both for free.

Despite the fact that I'm pretty handy with SQL it took me a LOT longer to write the no-Linq version of these tests and I'd much rather maintain those than the reverse.

None of the times I reported have anything to do with what actual applications with normal latencies and data logic will experience. In those cases the results show that you're likely to see little to no difference between using linq and not using linq (if you compile etc.) which is also a great result.

Overall I'm very pleased. I hope you will be too.

DLinq (Linq to SQL) Performance (Part 4)

ricom — Fri, 06 Jul 2007 03:32:00 GMT

Well it's high time I gave you some numbers for the new stuff.

In the original benchmark the Linq version was running at 13.62% of the original time. And while I'm discussing that result, Sekiya Sato pointed out an error in my original benchmark (see the comments of the above posting) in which I had one of my ISDBNull() checks backwards. That error made the "nolinq" version actually run 3.6% faster than it should have. So the number I reported, 13.62% should have actually been 14.09% -- let me restate that result for clarity, in May 2006, DLinq was running at 14.09% of the underlying provider speed in this (harsh) test case on my hardware and not 13.62% as previously reported.

I have in my hands a nice fresh build, which is similar to what you're going to get when you adopt Beta 2. The results below include my original test plus an some quick insert and update tests I added -- I'll describe those in the next installment. What we want to talk about right now is the select cases. The regular select is as orignially described. The syntax for the compiled select (and this really builds) is this:

var fq =CompiledQuery.Compile(
    (Northwinds nw) =>
        from o in nw.Orders
        select new OrderDetail
        {
                OrderID = o.OrderID,
                CustomerID = o.CustomerID,
                EmployeeID = o.EmployeeID,
                ShippedDate = o.ShippedDate
        }
);

Note that with the nice type inferencing you never have to see the generic types in your code but it's still strongly typed. To use this query you simply

foreach (var detail in fq(nw))
{
sum += detail.OrderID;
count++;
}

Now let's have a look at the numbers:

	no linq			with linq
	select	update	insert	original select	compiled select	update	compiled update	insert
run 1	915.25	4.87	4.29	497.81	858.07	20.65	21.05	20.25
run 2	916.25	5.02	4.76	491.59	864.60	20.34	20.62	12.02
run 3	942.86	4.87	4.66	496.57	859.11	21.03	20.47	16.08
average	924.79	4.92	4.57	495.32	860.59	20.67	20.71	16.12

The units for all of the above are test iterations per second so bigger is better.

	dlinq	nolinq	ratio
select	495.32	924.79	53.56%
compiled select	860.59	924.79	93.06%
update	20.67	4.92	420.19%	(DLinq is faster)
compiled update	20.71	4.92	421.00%	(DLinq is faster)
insert	16.12	4.57	352.66%	(DLinq is faster)

Wow that's pretty good. If you do nothing to your code, just raw internal improvements go from 14.09% of the underlying provider to 53.56% -- that's a 3.8x improvement. But look at what you can do with compiled queries: if you compile the select statement I got 93.06% of the underlying providers raw speed -- that's 6.6x faster than what I got back in May of 2006. This is a truly great result because, as I've mentioned before, this is a harsh test. With the normal overheads associated with actual business logic and data transfer this result basically means that you may not even be able to measure any throughput degradation at all if you use compiled DLinq queries in your program.

I think I'll let Matt talk about the details of how we did this because he did the work but I can give you the high level points if you haven't already guessed them from the previous postings

create custom methods that bind the data perfectly using lightweight code generation
create reusable SQL with parameters to avoid generating the SQL query again
provide read-only contexts to avoid any unnecessary entity management (this not needed anyway in my case because I new up an OrderDetail object with only part of the data)

When I modelled this on paper last summer it looked like we could get about 95% of the underlying provider speed plus or minus measurement error and we seem to have landed at 93%.

Now you may ask, why is DLinq doing better at updates than my code that writes directly to the underlying provider? I'll talk about this a bit more next time but the short answer is this: the code I wrote to do the updates looks like pretty typical SQL sent to the database in batches. However I didn't go to the trouble of creating prepared statements for update and insert cases, DLinq gives you this automatically. So despite my more complicated logic the savings DLinq got from superior SQL trumped my techinque.

And lastly, as always, this doesn't necessarily translate to any specific numbers for your application but it sure bodes well. I'm very pleased indeed.

DLinq (Linq to SQL) Performance (Part 3)

ricom — Fri, 29 Jun 2007 19:01:00 GMT

I’d like to start with a little housekeeping. Some readers asked me how I made the nifty table in part 2 that showed the costs broken down by major area.

It was actually pretty easy to create that table using our profiler. I did 500 iterations of the test case in sampled mode and that gave me plenty of samples. I could see which callstacks ended in mscorjit.dll – even without symbols -- which I never bothered using -- that gave me a good idea how much jit time there was. I could see the data-fetching functions from the underlying provider being called -- the same ones that appear in the no-linq version of the code (GetInt32, GetString and so forth) so I knew what the costs of actually getting the data were. I could see the path that creates the expression tree for the query and I could see stub dispatch functions. So I added up the related ones, broke it into 5 categories and then showed one more line for the bits that didn’t fit into any of those categories. Then I scaled the numbers up so that the part of the benchmark I cared about was 100% (there was other junk in my hardness not relevant to the benchmark). That’s it :)

One more gotcha though, when I talked to Matt about this we realized that I had reported the breakdown from an internal build that included one change he had made for me from the May 2006 CTP. The breakdown you would see if you did this experiment again, on a May 2006 CTP build, would have reflection costs instead of jitting costs. I'll discuss that more when I go into the specific improvements we made which are coming soon...

But, meanwhile, let's move on to the real topic of Part 3…

Per Row Costs

In this area there were two things that I worried about.

Entity/Identity Managment

The data binding problem used to be at the top of my list but entity creation costs started to worry me more. This query is sort of an example of a troublesome one:

var q = from o in nw.Orders
select o;

Doing foreach over the above is much less efficient than the below, and the astonishment factor for that might be pretty high.

var q = from o in nw.Orders
select new {o.everything …};

In general all these “read a bunch of data” queries are likely to suffer significant overhead because of temporary allocations and the mid-to-long life of the entities being enumerated. In the first formulation, the objects have to be stored because they might be modified as you foreach over them (or later). In the second formulation no object management is required because you’re newing up a synthetic object not associated with a table.

But, there is a strong expectation that code is just a forward-only read operation with at most one temporary business object created per iteration. Especially if you don’t modify the objects in the foreach loop is read-only. That expectation also implies a nifty solution.

Since the Connection is a property of the context you could reasonably have multiple contexts associated with one connection. In particular a read-only context could be just the thing to avoid all this entity creation. Importantly the read-only context can be connected to the same Connection which means isolation concerns go away – it looks like one logical view of the database. This is important because otherwise a thread owning two DataContext objects could self-deadlock.

So I recommend creating some kind of read-only context to avoid the overhead of object management when it was not necessary.

Data Binding

The second item is overhead associated with extracting the data. In the then current implementation there were several virtual calls between the MoveNext/GetCurrent on the enumerator and the actual data field fetches.

However, if we do query compilation as described in the last installment we have the opportunity to significantly limit the dispatches – as low as one delegate call.

To do this we need to pre-build the necessary helper for getting and storing the fields so that there is effectively straight line code calling the underlying provider. Basically the same code you would have to write if you were doing the data access manually.

Pre-compiling it also means we don’t have to jit the helper every time the query runs – we have a place to store the compiled helper function that stores the data.

At this point we had a pretty good idea what kinds of actions to take to make things a whole lot better.

DLinq (Linq to SQL) Performance (Part 2)

ricom — Tue, 26 Jun 2007 03:10:10 GMT

So after getting some high level times I started digging into the particulars of the costs more broadly and I ended up studying a very simple query like the below one.

Northwinds nw = new Northwinds(conn);

var q = from o in nw.Orders
           where o.OrderId == orderid
           select o;

foreach (Orders o in q)
{
    …
}

It was the per-query costs that seemed to be the greatest trouble spot in the then-current profiles. Those costs would be the most problematic for complex queries which return comparatively few rows – all too common in business logic. For small numbers of rows the rough bucketization of costs looked like this:

Category	Time
Total Benchmark	100.00%
Query Build	24.40%
Query Enumeration	74.55%
Dispatch Glue	7.34%
Jitting Costs	18.07%
Data Reading	49.14%
Misc	1.06%

In short the problem is that the basic Linq construction (we don’t really have to reach for a complex query to illustrate) results in repeated evaluations of the query if you ran the query more than once.

Each execution builds the expression tree, and then builds the required SQL. In many cases all that will be different from one invocation to another is a single integer filtering parameter. Furthermore, any databinding code that we must emit via lightweight reflection will have to be jitted each time the query runs. Implicit caching of these objects seems problematic because we could never know what good policy is for such a cache – only the user has the necessary knowledge.

But all is not lost... the usual parameterized query model seems to be helpful here without unduly complicating everything. You could imagine a sequence something like this:

Func<Northwinds, IQueryable<Orders>, int> q =
        CompiledQuery.Compile<Northwinds, int, IQueryable<Orders>>
                ((Northwinds nw, int orderid) =>
                            from o in nw.Orders
                            where o.OrderId == orderid
                            select o );
Northwinds nw = new Northwinds(conn);
foreach (Orders o in q(nw, orderid))
{
     ...
}

The important thing here now is that q is a durable thing that can be applied to different data contexts and we've identified the orderid paramater. You'll have to forgive my syntax I don't think there's a compiler in existance (old or new) that compiles precisely the above but hopefully you'll get the idea.

Importantly, upon compilation, the query can be reduced to some kind of prepared statement. At this time any helper methods that need to be code-generated are also created. Upon binding we do the minimal query formatting for the constants and no jitting. The compiled query has lifetime specified by the user, so it lives exactly as long as it needs to.

These operations would drastically reduce per query overhead while simultaneously giving us a good place to hang state with suitably lifetime – compiled queries.

That seemed to get us forward progress on the per-query costs but what about the per-row costs?

We had a couple of different ideas to help with those as well.

Stay tuned for part 3. :)

P.S. Keep your eye on Matt Warren's Weblog as he'll likely comment on what I'm saying as the series evolves, it was his hand that actually made the changes I'm talking about here.

Caching Redux

ricom — Mon, 25 Jun 2007 18:47:52 GMT

I got some interesting questions about how to build good middle-tier caches in my inbox last week. I cleaned up the responses a little bit and I'm posting them here because they're actually pretty general. I've written about this before but some things merit repeating :)

Here's what I wrote:

If I had a dime for every person who thought caching was the answer but then didn’t actually build a cache…

First, consider your cache policy carefully. As I’ve often written, caching implies policy

http://blogs.msdn.com/ricom/archive/2004/01/19/60280.aspx

And as I told Raymond – a cache with bad policy is another name for a memory leak

http://blogs.msdn.com/oldnewthing/archive/2006/05/02/588350.aspx

Raymond turns this into some excellent recommendations, including instrumentation and observation which result in cache design by a quantitative approach.

If I had a dime for everyone who built a cache because they thought it was a good idea but then did not measure the efficacy of what they had built…

Explore the space, try rough experiments at different layers and try different policies. Often very aggressive policies (fast retirement of cache data) are effective but you must understand not only how data gets in the cache (that is obvious) but how does it get OUT? Actively or passively? Based on limits or hit rate or?

Whatever you do, be sure you do it on the basis of measurements. Any kind of automatic “magic” caching layer that somehow knows about new business objects immediately sounds like a disaster to me. It’s not a question of knowing the business objects it’s a question of knowing usage patterns and policy. I don’t know how to do that automatically -- but maybe your particular problem has patterns you can leverage. I also know that (e.g.) SQL server already has a good cache at the data level and if you do your job right that is often all you need.

Policy is everything.

DLinq (Linq to SQL) Performance (Part 1)

ricom — Fri, 22 Jun 2007 22:20:00 GMT

[ By popular demand, here are links for all 5 parts in the series Part 1, Part 2, Part 3, Part 4, Part 5
-Rico ]

I love Linq. Really. That might scare you because it has all these wacky new constructs and as a performance guy you'd think that I'd be all about getting rid of abstractions and just getting to the metal. But don't be scared, I haven't lost my mind. Linq is great because, even though it adds some levels of complexity, it simulataneously increases the chunkiness of the work that the framework receives in such a way that it creates fantastic opportunities to deliver performance. Just like SQL can do a great job optimizing database queries because they are chunky enough.

And speaking of databases, DLinq is really where the opportunities for amazing coolness are present.

I first started looking at the performance of DLinq (it's officially called Linq to SQL but I still call it DLinq)shortly after the May 2006 CTP -- the very same one many of you are still using. There were some great opportunities at that time and I'm happy to report that we've capitalized on a lot of what we found then, in fact I'll be writing about that in the next few postings. But for today I want to talk about how things looked back in May of 2006. How they might still look to you at this very moment if you're using the May 2006 CTP.

I wanted to look at the basics of DLinq performance in a very simple case to get an idea what the raw overhead was. So I set up a harsh test environment:

northwinds database, local
many queries have already been run so the database is hot, no disk activity
the body of the dlinq query is minimal so all that code is hot
no enties need be stored, so CLR memory is also hot

Why would I do this? Because it's the "nightmare" scenario for DLinq. There is no business logic. There is no database latency. There is no inherent CLR overhead associated with the processing. The cost is DLinq and nothing but DLinq. Normally there is some business processing so the DLinq code is only a portion, maybe even a small portion of the total processing time. Normally you have to connect to a database over the network and maybe even read the data off disk, again that reduces the portion of time you spend waiting for DLinq. But not in my test case... my test case is beating on DLinq so I can see how it stands up.

I won't show you the whole program (you could create it in a few seconds with the CTP) but the key parts are like this:

Here's the query

var q = from o in nw.Orders
            select new OrderDetail {
                OrderID = o.OrderID,
                CustomerID = o.CustomerID,
                EmployeeID = o.EmployeeID,
                ShippedDate = o.ShippedDate
           }

Then we run this in a loop

foreach (var detail in q)
{
sum += detail.OrderID;
count++;
}

Note the highly useful (sarcasm) business logic -- adding the OrderId. I did that so I could print the total at the end and make sure as I change the test that I was really reading them. Same for the count. Any real application would do something -- anything -- with those order detail lines, my application is basically throwing them away.

OK great so I have a little test harness scientifically designed to maximize the DLinq overhead. What do I compare it against?

Well I wrote this other program that gets the same data and does the same (trivial) operation the old fashioned way so that I could compare.

The body of that one looks like this:

SqlCommand cmd = con.CreateCommand();
string cmdText = "select OrderID, CustomerID, EmployeeID, ShippedDate from Orders";
cmd.CommandText = cmdText;
SqlDataReader dr = cmd.ExecuteReader();
while (dr.Read())
{
    OrderDetail o = new OrderDetail();
    o.OrderID = dr.GetInt32(0);
    o.CustomerID = dr.GetString(1);
    if (!dr.IsDBNull(2)) o.EmployeeID = dr.GetInt32(2); else o.EmployeeID = null;
    if (!dr.IsDBNull(3)) o.ShippedDate = dr.GetDateTime(3); else o.ShippedDate = null;
    sum += o.OrderID;
    count++;
}

It's a lot more code but that's how you do it using just SqlDataReader.

So now I can measure how many queries/sec I can do with the DLinq code and compare it to the SQLDataReader equivalent.

Any guesses? DLinq can't be faster because of course it uses SqlDataReader itself to do the job so the best you could get is a tie.

No peeking now.

Think about it. How much slower do you think DLinq was in May of 2006 in this "worst case" scenario.

Got your number?

Last chance now.

Final answer?

OK here's what I got when I did the experiment, back in July of 2006.

Build/Test	Time for 500 Queries	Queries/sec
May 2006 CTP	8.027s	62.29
Raw Cost (SQLDataReader)	1.094s	457.04

You can pretty much eyeball it from those times. In May 2006 DLinq is running at about 1/8 the speed of the underlying provider (13.62%).* **

We can do better than that. And we did...

Stay tuned for the details and some modern era DLinq results.

*Remember no real application would ever see a result as poor as 13.62% because of course they would be doing "actual work" as well as the DLinq operations resulting in more comparable performance.

**Sekiya Sato (see below) pointed out an error in my original benchmark in which I had one of my ISDBNull() checks backwards. That error made the "nolinq" version actually run 3.6% faster than it should have. So the number I reported, 13.62% should have actually been 14.09% -- let me restate that result for clarity, in May 2006, DLinq was running at 14.09% of the underlying provider speed in this (harsh) test case on my hardware and not 13.62% as previously reported.

Optimistic vs. Pessimistic Locking consequences

ricom — Thu, 24 Jun 2004 19:39:00 GMT

Putting on my "database guy" hat, a few days I was asked to comment on locking techniques for databases and to point out some problems with using transactions as a pessimistic locking technique. Here's what I said...

Some definitions:

Optimistic locking -- that's where you assume things will go well and design your locks so that they handle conflicts as the exceptional case

Pessimistic locking -- the converse where you assume conflicts are likely and create some kind of reservation system where sections are locked while they are edited

[In response to the suggestion, you must] remember that transactions are not really durable things and that any strategy that is designed around transactions having very long life (e.g. minutes) must also be designed around the chance that such a transaction is aborted for any number of reasons, not the least of which is deadlock.

The math of the situation tells you that the longer running a transaction is the lower the likelihood you can commit it.

So using something as simple as open transactions for your locking strategy is pretty much right out.

Now the moment you choose a scheme where a given client applies a (pessimistic) software lock on some set of objects you have another set of issues -- what if the client disconnects? How to you administer and recover these locks? What operations do you allow on the locked objects? How will you provide (manually) the isolation that's needed (if any) to hide any in-flight changes that are committed from the database perspective but pending based on business rules?

An optimistic locking strategy provides one omnibus solution to these problems. Other solutions are possible but you should not underestimate their difficultly because they create interim/valid committed states. States you might find yourself in, for instance, after a backup, with no supporting client info what-so-ever.

It's not that the pessimistic lock is inherently bad but once you've decided on such a path, you then have a variety of other problems to solve. By its nature optimistic locking does not create interim valid states and so finesses many of these issues.

It turns out many of these same issues apply to pretty much any free threaded datastructure.

Some thoughts/advice about databases and caching

ricom — Mon, 10 May 2004 19:57:00 GMT

As usual, kindly allow me to speak in rough terms so that I might be brief keeping in mind that there are exceptions to what I write below.

I think the most important thing to remember about serving data from a database is that, as a practical matter, you can’t realistically serve data that you know to be absolutely current. Even if you somehow get the data from the database and magically format it up in zero time there are enough delays along the way that it might have changed by the time your end user sees it and tries to do something with it.

Your user is going to be seeing stale data, the only question is, how stale.

Not only is the data going to be stale, it’s also likely to lack self-consistency. For instance, unless you go to very great pains (and cost), when your user asks for “page 2” of a dataset via the “give me page 2” button most algorithms for computing “page 2” might return duplicate items, or might miss items entirely if something is added or removed that would have been on “page 1”.

Yet despite these theoretical and practical limitations, real people interact with real databases every day and have satisfactory experiences. This comes about because in practice good database analysts think carefully about the sorts of queries that will be made against their database, and the elementary business operations that will change the contents of that database, plus the nature of the user interface that will be doing the asking.

Now at last we come to the performance part of this discussion :)

While you’re thinking about the sort of experience that you want your user to have, it’s vital to consider what sort of tolerance you can afford to staleness and correctness and how to best leverage that design to get the best overall experience (including performance) for your customers.

I’m sure many people are screaming about trading off correctness for performance at this point but I’ll remind you that delivering 100% correct data to your customers is impossible as a practical matter – nobody could afford such a system. So from there good engineering forces you to think, “Since perfection is far too expensive, what guarantees should I be making?”

At the core of the database the kinds of consistency guarantees that you want to make will translate into factors like the size of your transactions and the breadth of allowed queries. Having made those choices you are then in a position to select a suitable isolation level to facilitate them.

The choices you make in the next layer, your business objects, tend to mirror the choices made at the database level. At this level you might concern yourself with caching choices to avoid a lot of round-trips to the database. Avoiding round-trips is what caching is all about.

Some things to consider:

Every caching strategy is based on not asking the database a question it otherwise would have had to answer which ultimately increases the staleness of the data you serve. Is a time-based staleness guarantee suitable for your application? This approach is actually a lot more versatile than it might seem.

What cache-hit ratio do you need to have for your cache to be a net win? Remember SQL itself caches the underlying data which partly competes with any caching you might do yourself. Given the mix of queries you expect to arrive at your business layer, what hit rate would you need to actually see a benefit?

Caching is no substitute for a well considered indexing strategy on the server itself. Enormous caches should be the province of the database, their presence elsewhere often indicates that not enough attention has been paid to the schema and indexes.

Is it fairly easy to detect when your cache is “too stale” or does that require a complex computation? If it requires complex calculations you might want to try something simpler, or you might want to try reversing the logic so that the server notifies you of potential changes – but beware, complicating the back end might add friction to the whole system making it very difficult to do large batches of updates for instance. Complex notification systems are frequently not worth the effort or the expense and of course they do not necessarily provide a 100% up-to-date data guarantee either.

One caching policy is not likely to be suited to all your applications needs. For instance, you can probably have an astonishingly large time tolerance on data you use for things like converting from a “State ID” to a “State Name” but I could hardly recommend the same for information associated with an “Order ID.”

A cache that holds “partly cooked” data which can be assembled or otherwise quickly processed to create “fully cooked” results is often preferable. The partly cooked results can be used in a greater variety of queries and often take less space, increasing the effective size of the cache. In web servers this means caching fragments of pages, or caching the raw data from which the HTML can be quickly formatted, in preference to caching entire HTML pages.

If a cache will be a key feature of your business layer be sure to optimize the underlying schema so that it is suitable for the cache populating queries rather than potentially optimizing for accessing singleton objects. This is akin to designing DRAMs so that they are best at filling the CPU's cache lines and not optimizing for byte-wise access.

Summary

Select a simple caching mechanism that produces a predictable and satisfactory customer experience, does not add much in the way of complexity, and co-operates well with the underlying schema.

Hold the mayo :)