SQL Server Compact - Compact & Capable : Performance

After moving the database from one platform to other, the first SqlCeConnection.Open() takes more time

SQLCEBLOG — Wed, 01 Apr 2009 15:16:00 GMT

If you move a SQLCE database from one platform to other, it's first SqlCeConnection.Open() takes more time and also increases the database file size.

The Reason: For an index on string type columns, SQLCE uses LCMapString API to get the normalized sort key. LCMapString API behavior will be different for different NLS sort versions. NLS sort version is different for Desktop Windows OS and WM/WCE. During the first database connection open, SQLCE rebuilds all indexes if there is a NLS sort version mismatch with the last accessed OS platform. Index recreation requires space on database file and hence the database file size will be increased. You can use Compact()/Shrink() API to get the database file with approximately old file size. Because of this index recreation, first database connection open takes more time as it has to rebuild all indexes.

Note: If your database doesn’t have any indexes you will not see above mentioned symptoms on first connection open.

Don’t inter-op SQLCE 3.0 databases between devices and desktop:

SQLCE started supporting desktops only from SQLCE 3.1 (3.0.5300.0) release. SQLCE 3.0 doesn’t have index rebuilding logic and the index re-creation was added during SQLCE 3.1 (3.0.5300.0) release as it is required to support platform inter-op for SQLCE databases. On desktop you can open SQLCE 3.0 databases, which were created on devices. Again If you open these databases on SQLCE 3.0 devices, your device app might receive an error/exception.

For example: Opening a SQLCE 3.0 database with SSMS will use SQLCE 3.1 bits on desktop and hence it would drop and recreate index data. When this file is copied back to device, index data may not be valid and receive an error/exception.

Namaskaram!

Manikyam Bavandla

Sync Services optimizations

SQLCEBLOG — Mon, 06 Oct 2008 15:40:00 GMT

There are some optimizations that can be done for an "Sync Services" sync scenario. These are, (might not be an exhaustive list)

Small sync chunks (using smaller SyncGroups which are only absolutely necessary)
Index creation on tracking and filtering columns
Transfer encryption of data payload on web
Batching the changes
Filtering the data at various stages to reduce network load

    Sync (synonym for "Sync Services sync", in this post) interfaces with the providers using DataSet objects. That is, sync uses DataSet to store and retrieve changes happening at client and server. DataSets are known to be not very memory efficient, and the amount of memory consumed by these, is a bit of a concern when one is using these in device clients in Sync Services.

    Sync happens in chunks, using the SyncGroups that one supplies. That is, all data in a SyncGroup is logically one sync, and corresponds to one DataSet. Since sync DataSets are one per SyncGroup, and these are needed only till the corresponding SyncGroup commits, we can optimize the memory taken by DataSets, using smaller SyncGroups. Divide sync tables into various SyncGroups, so that, each group has it’s own (smaller) DataSet, and it is disposed (by garbage collector) once the SyncGroup changes are committed. However, to ensure logical consistency, you should always keep the tables which are related (by a foreign key constraint, for example), in the same SyncGroup.

Example:
Tables Order and OrderDetails are related.
Tables Customers and ShippingDetails are related. But, these are unrelated to the first set.

SyncGroup customerSyncGroup = new SyncGroup("Customers");

SyncTable customerSyncTable = new SyncTable("Customer");
customerSyncTable.CreationOption = TableCreationOption.DropExistingOrCreateNewTable;
customerSyncTable.SyncDirection = SyncDirection.DownloadOnly;
customerSyncTable.SyncGroup = customerSyncGroup;

SyncTable orderShipSyncTable = new SyncTable("ShippingDetails");
orderShipSyncTable.CreationOption = TableCreationOption.DropExistingOrCreateNewTable;
orderShipSyncTable.SyncDirection = SyncDirection.DownloadOnly;
orderShipSyncTable.SyncGroup = customerSyncGroup;

SyncGroup orderSyncGroup = new SyncGroup("Orders");

SyncTable orderSyncTable = new SyncTable("Order");
orderSyncTable.CreationOption = TableCreationOption.DropExistingOrCreateNewTable;
orderSyncTable.SyncDirection = SyncDirection.DownloadOnly;
orderSyncTable.SyncGroup = orderSyncGroup;

SyncTable orderDetailSyncTable = new SyncTable("OrderDetails");
orderDetailSyncTable.CreationOption = TableCreationOption.DropExistingOrCreateNewTable;
orderDetailSyncTable.SyncDirection = SyncDirection.DownloadOnly;
orderDetailSyncTable.SyncGroup = orderSyncGroup;

    Rather than having a single SyncGroup with all 4 tables, one can create two sync groups. One for the first set, and the second for the other set.

    Also, to increase the performance of enumerating process at server, one can create indexes on the tracking and filtering columns. As an example, let us say one has two tracking timestamp columns __CreateTime and __UpdateTime. Since, our server side changes enumerating queries always have filtering based on these columns, one should create indexes on these, so that, enumerating server changes becomes faster, and, sync performance increases.

    In case of device syncs, transferring the data on the network/internet is also going to cost more time/resources. Since, transfer of data from a web service/server machine to a device is time/bandwidth consuming, one should look at compressing data sent/received. This is mostly the case, when data being synched is large (Something like, initial synching SQL Compact with SQL Server). One can use System.IO.Compression or your own implementation of compression/decompression algorithm. (More details at http://msdn.microsoft.com/en-us/library/system.io.compression.aspx). Another blog post that refers to this issue and proposes solutions is at, (http://blogs.msdn.com/mahjayar/archive/2008/10/01/dbsyncprovider-improving-memory-performance-in-wcf-based-synchronization.aspx)

    Sync Services Server side supports batching. Please refer to http://msdn.microsoft.com/en-us/library/bb902828.aspx for more details.

    Sometimes, we may have server tables holding and synching data, that is alien to clients. For example, server table has a large GPS map of some place (in custom format) per row, that is associated with the data in other columns in the row. The map is stored as an image at client side. But, since, the data is of proprietary format, client can’t interpret the data, and it is not used anyway at client side. And also, since, it is large data, lot of network bandwidth is consumed and response time is compromised for transferring this across. It is better, if one can filter these columns out at server side, rather than sending to client.

    Another scenario is as follows: The client devices of a Sync application are used by Salesmen of a company, and the server side is the company office, where data pertaining to each salesman is stored in a SQL Server database. When the client devices sync, they are mostly concerned about the data relevant to them alone, and not to other sales persons. Here also, one needs to filter rows from the table.

    To facilitate filtering of rows/columns at client/server sides, one can use a number of strategies. For filtering data sent from client (or at the client side), one can use the callback architecture of "Sync Services". For server side, since you have more control there, one can choose to strip down the DataSet, when one is sending it over the wire/use callbacks to strip off DataSet data/client can supply queries to server side adapters, which filter data.

Explaining each of these options:

Callback architecture of sync: Sync provides various callbacks during the process of Synching. One can hook in the callbacks for accessing/modifying the dataset generated at every stage of sync. For example, one can add/drop some columns to the dataset, after changee enumeration is done at client side. After changing the dataset, effectively, the changes being applied to client database are changed. The same provision is present at server side also.

This callback architecture can be used to modify the DataSet applied/sent to server, on the fly. This is one type of filtering, possible at both client and server.

Server side gives more options when sending data over the wire. First of all, server chnages are enumerated within the "Sync Services" application itself. That is because, server change enumeration is done by queries supplied by the Sync Services client. So, which columns to select from server, can be controlled from the client side, when specifying queries like, SelectIncrementalInserts etc... This is another way of programmatically filtering server data.

One more option to filter data in a N-tier architecture is that, filtering the wire transferred content. In an N-tier architecture, client and server are in two different, but connected machines. Typically, client machine has a proxy server provider, which delegates calls to enumeraion/application of changes etc... to a web service. Web service talks to a backend data store, and services the data and operation requests. In this setting, interesting possibilities like, use of compression over the wire, and filtering over the wire etc... come up. When server side data store gives data, it can do some filtering. Also, the client side proxy can be used to filter irrelevant data. This is another type of filtering.

Example for callback architecture:
For client side DataSet filtering (client side), use SelectingChanges before the enumeration and ChangesSelected after the enumeration
For server’s data set filtering (client side), ApplyingChanges before the application of changes and ChangesApplied after the application
For filtering server side changes (at server side):SelectingChanges before the enumeration and ChangesSelected after the enumeration
For client’s data filtering (at server side):ApplyingChanges before the application of changes and ChangesApplied after the application

In every one of these methods, eventArgs.GroupMetadata and eventArgs.Context.DataSet are the ones need to be changed.

Thanks

Udaya Bhanu,

SDE II, SQL Compact

Query performance

SQLCEBLOG — Wed, 30 Apr 2008 08:15:00 GMT

Understanding the reason for slowness of a SQL query and then tuning to boost it is a slightly complicated process, but it can be extremely rewarding in some cases. In this post, I am going to list down some ways in which you can tune the query performance for SQL Server Compact, along with pointers to relevant online resources. If you are a seasoned database developer, you probably know most of the tricks already.

Before I jump into it, an overview of the basics first.

Basics:

Displaying actual and estimated Execution plans for SQL Server Compact: You should be familiar with Execution Plans (also called as query plans and showplans).

For non-parameterized queries: There are several easy ways to do this

When database is on a docked device or desktop: Generate a query plan using SQL Server Management Studio (SSMS) that gives a nice graphical view of the query plans. (You can connect to the database on device as long as it is docked. No need to copy the file to desktop.)
When database is on a device: Generate a query plan using Query Analyzer. This generates a .sqlplan file that can be viewed graphically in SQL Server Management Studio as before (preferable) or opened as a plain XML file.

For Parameterized (as well as non-parameterized) queries:

Use the TSQL Statements SET SHOWPLAN_XML (for estimated execution plan) and SET STATISTICS_XML (for actual execution plan), followed by SELECT @@SHOWPLAN to generate the query plans. Store the results as an XML file. (Tip: To view the query plans graphically, just rename the file with a .sqlplan extension and it is ready to be opened in SSMS like an ordinary non-parameterized query plan.)

Indexes and statistics on indexes:

One of the important tasks of the query optimizer is to select the right index to execute the query. An optimizer can make better decisions if it has histogram data (i.e. statistics) about the distribution of values for an index.

CREATING/UPDATING/DELETING statistics on an index: SQL Server Compact 3.5 creates statistics on all indexes by default and updates them as required. (You can surely tweak these settings, but it is advisable not to. See CREATE STATISTICS /UPDATE STATISTICS/DROP STATISTICS).
To view the statistics distribution, the built in stored procedure sp_show_statistics is pretty handy.
Statistics were not created by default on primary key indexes till version 3.1, so you might want to check this and add them yourself :- ).

... A lot of other things too J, but this will do for the time being.

The Meat:

If you are trying to improve the performance of your application that uses SQL Server Compact, make sure you have checked the following resources on MSDN already (In this post I will not go into any of them):

Database Design and Performance: This page talks in some detail about the following:

Database denormalization
Variable and fixed sized columns
Effect of row and key lengths

Query Performance Tuning: This page explains some of the following:

Improving indexes by creating selective indexes, choosing the columns in a multi-column index, issues in indexing a small table etc.
When to create indexes
Using parameterized queries where possible.
Rewrite queries where possible
Using base table cursor (CommandType.TableDirect) mode for simple queries.

(The article on Query Process Overview and Performance Tuning approaches for SQL Server CE 2.0 covers the same points as above, but I found it better than the MSDN links J. It is relevant for SQL Server Compact 3.5 too.)

Now here are some additional ways you can improve performance:

1. Recompile Stale query plans

<StoryTime>Sometime back I got a complaint about slow performance of our database even though the user was using parameterized queries and had created the correct indexes everywhere. The sequence in which he was performing operations was something like this:

- Create the full database schema (tables, indexes etc).

- Prepare SqlCeCommand objects for *all* queries that will ever be used by the application.

- Populate the database with actual data. Until now all tables were empty.

- Execute the queries using previously prepared command objects

What's wrong with this?

A prepared command object (SqlCeCommand) holds a query plan. The query optimizer generated this query plan and had optimized it for the state of database when the plan was generated. In this user's case, the database had just empty tables at that point in time. The optimizer therefore inferred that using a table scan is the best way of executing the query (as opposed to using any index). As a result, all his query plans used just table scans instead of the right indexes even when the table size increased!

</StoryTime>

Moral of the story: Caching query plans is good, but only as long as the state of database when they are compiled is representative of the average state of database during execution. If your data is changing rapidly, it is better to recompile the queries every once in a while.

So, beware of stale query plans!

Note that query plans can be cached for both parameterized and non-parameterized queries and you can run into this problem in either case. (Another insider hack: SqlCeCommand.Prepare doesn't really prepare the query plan. It just marks the command object for plan (re)compilation and the plan is compiled when SqlCeCommand.ExecuteXXX is called next time.)

2. Impression of using parameterized queries without really using them

Question: What's wrong (rather sub-optimal) with the following code:

SqlCeCommand cmd = con.CreateCommand();

cmd.CommandText = "SELECT * FROM table1 WHERE C1_DECIMAL = @p1";

cmd.Parameters.Add("@p1", SqlDbType.Decimal);

cmd.Parameters["@p1"].Value = 3.5;

SqlCeDataReader rdr = cmd.ExecuteReader();

while (rdr.Read()) { /* read the data */}

rdr.Close();

cmd.Parameters["@p1"].Value = 335.01;

rdr = cmd.ExecuteReader();

while (rdr.Read()) { /* read the data */ }

rdr.Close();

Answer: The query plan gets compiled twice, once for every ExecuteReader call!

Reason: First time the plan is compiled, the parameter's precision is 2 and scale is 1 (as inferred from its value ‘3.5'). The query plan therefore uses this precision and scale values. The only other values it can accept are those that fit within this range.

When the query is executed again, the parameter's precision is 5 and scale is 2 (as inferred from its value ‘335.01'). Since it doesn't fit in the parameter as inferred during first plan compilation, the plan is recompiled silently!

Solution: If you can anticipate the range (size, precision or scale) of the parameter values, then specify it explicitly. The plan will be generated based on the specified range then. This holds true for all character, binary and numeric types. (The flip side is that for any parameter value that does not fit into the specified range, an error will be thrown.) So the above code can be modified as follows:

SqlCeCommand cmd = con.CreateCommand();

cmd.CommandText = "SELECT * FROM [TABLE1] WHERE [C1_DECIMAL] = @p1";

cmd.Parameters.Add("@p1", SqlDbType.Decimal);

cmd.Parameters["@p1"].Precision = 10; // Playing safe by taking a larger range!

cmd.Parameters["@p1"].Scale = 5;

cmd.Parameters["@p1"].Value = 3.5;

SqlCeDataReader rdr = cmd.ExecuteReader();

while (rdr.Read()) { /* read the data */}

rdr.Close();

cmd.Parameters["@p1"].Value = 335.00;

rdr = cmd.ExecuteReader();

while (rdr.Read()){ /* read the data */ }

rdr.Close();

3. Query optimizer did not choose the best index

The query optimizer usually does a good job of choosing the index. However, it does it on a best effort basis and there can be cases where it doesn't pick the right index. In such cases, it makes sense to use index hints and outsmart the optimizer:

E.g. ...FROM [TABLE1] AS T1 JOIN [TABLE2] AS T2... can be rewritten with index hints as

...FROM [TABLE1] AS T1 WITH (INDEX(IX_On_Table1))

JOIN [TABLE2] AS T2 WITH (INDEX(IX_On_Table2))...

How do you know if your index is indeed better than the index chosen by optimizer?

Well, the easiest and brute force method is to run the query with and without an index hint and see which is better. I often use the Query Analyzer on device and SSMS on desktop to get the execution times. Another way is to generate query plans for both queries and study them for better predicate matches, index ordering etc.

4. Query optimizer did not choose the best join order

As in the case of choosing a wrong index, an optimizer can sometimes choose a wrong join order too. Again a corner case, but not impossible to run into. In such cases, you can specify the join order explicitly:

E.g. ...FROM [TABLE1] AS T1 INNER JOIN [TABLE2] AS T2 ON ... can be rewritten with a forced join order as

...FROM [TABLE1] AS T1

INNER JOIN [TABLE2] AS T2 ON ...

OPTION (FORCE ORDER)

As with any sort of query hint, you should be extra careful that you are doing the right thing. For instance, the join order chosen for a query can change between multiple runs due to several reasons. The cardinality of the join tables could have changed significantly, or the data distribution could have changed. So even though the join order that you selected at the time of deployment was optimal, it may not remain so all the time. Ordinarily the optimizer would decide the join order on fly and therefore can adjust to such changes.

Not so when the join order has been forced.

That's all for the time being.

-Pragya Agarwal