SQL Server Compact - Compact & Capable : Engine

After moving the database from one platform to other, the first SqlCeConnection.Open() takes more time

SQLCEBLOG — Wed, 01 Apr 2009 15:16:00 GMT

If you move a SQLCE database from one platform to other, it's first SqlCeConnection.Open() takes more time and also increases the database file size.

The Reason: For an index on string type columns, SQLCE uses LCMapString API to get the normalized sort key. LCMapString API behavior will be different for different NLS sort versions. NLS sort version is different for Desktop Windows OS and WM/WCE. During the first database connection open, SQLCE rebuilds all indexes if there is a NLS sort version mismatch with the last accessed OS platform. Index recreation requires space on database file and hence the database file size will be increased. You can use Compact()/Shrink() API to get the database file with approximately old file size. Because of this index recreation, first database connection open takes more time as it has to rebuild all indexes.

Note: If your database doesn’t have any indexes you will not see above mentioned symptoms on first connection open.

Don’t inter-op SQLCE 3.0 databases between devices and desktop:

SQLCE started supporting desktops only from SQLCE 3.1 (3.0.5300.0) release. SQLCE 3.0 doesn’t have index rebuilding logic and the index re-creation was added during SQLCE 3.1 (3.0.5300.0) release as it is required to support platform inter-op for SQLCE databases. On desktop you can open SQLCE 3.0 databases, which were created on devices. Again If you open these databases on SQLCE 3.0 devices, your device app might receive an error/exception.

For example: Opening a SQLCE 3.0 database with SSMS will use SQLCE 3.1 bits on desktop and hence it would drop and recreate index data. When this file is copied back to device, index data may not be valid and receive an error/exception.

Namaskaram!

Manikyam Bavandla

Insight into SQL Server Compact database corruptions

SQLCEBLOG — Wed, 25 Jun 2008 18:20:00 GMT

Database corruptions are often the most painful bugs to repro and debug. A good percentage of the SQL Server Compact bugs reported are in this category. However it so turns out that in majority of the cases Compact is not the culprit. Surprised!! Let us see why.

How SQL Server Compact detects a file corruption?

Compact database comprises of a series of pages, each 4K in size. When a page is written to disk the following sequence of steps is executed.

1. Compute the checksum on the page data and store it as part of the page.

2. Encrypt the page.

3. Write to disk at a given file location or offset.

4. Update system pages/run time cache with the page offset.

When a page is read from disk the following sequence of steps is executed.

1. Read from the disk.

2. Decrypt the page.

3. Re-compute the checksum and compare it with the expected value. (If it fails page is not fully baked. Meaning file write failed)

4. Check if this is the same page that we are expecting as per the system pages/ run time cache. (If this fails we lost a complete page write)

If (3) or (4) of the above read steps fail, the db is marked as corrupt and any further db activity is suspended till it is repaired successfully.

Notes:

· Verify reports step (3) errors as “Bad Checksum - idPage: %d, tyPage: %d, iPage: %d”.

· Verify reports step (4) errors as “Page with valid ordinal not found - idPage: %d, iPage: %d”

What could have triggered the corruption?

1. A simple device on/off or suspend/resume can result in half-baked file. (No reason to panic as we handle this as described in the next section.)

2. Compact relies on underlying FAT/TFAT file system to manage the database file. Any bugs in these components can lead to Compact file corruption. Compact stresses file system in many different ways as it tries to keep the file size as small as possible. As it employs shadow paging the writes and reads are quite random. (Note: 6 months back a bug has been identified in FAT/TFAT system that results in a Compact file losing some of the clusters during shrink operation.)

3. If SD card is being used for db file storage, it’s being equally stressed as the above.

4. More importantly SD cards bring in additional complexities during Suspend/Resume cycles. Here is a good blog link: http://blogs.msdn.com/windowsmobile/archive/2007/01/12/everything-you-want-to-know-about-sd.aspx

Am I suggesting SQL Server Compact bugs can’t result in any kind of corruption?

Of course Compact code bugs can result in corruption. A lot of them have been found internally before v3.0 release and also reported by customers thereafter. They are more logical in nature like index constraint is compromised or a table page says there are 1000 rows when there are only 55. We did a very good job in fixing these and they are hard to find now. However Compact bugs don’t result in physical corruptions like “checksum errors” or “page missing” errors.

What does Compact do to prevent corruptions?

We believe most of the corruptions should be due to device suspend/resume or on/off cycles. For ex: Compact is trying to write to a file and the user simply boots his device. Now file is in half-baked state. Or let’s say process gets killed before it can complete the flush cycle. How do we avoid this?

Well, we use shadow paging to overcome this situation. We can loosely categorize the db pages into user data pages, system pages and header page. As the name suggests user data pages contain all the user data. Header pages and system pages help us locate a specific user page on the disk. When it’s time to flush the changes to disk, we write shadows of the user data pages first. Then the shadows of the updated system pages are written. Finally the header page is written. The old pages are retained till the header page is updated. This ensures that we have a transactional write to the disk. Either the change is completely present or not.

Conclusion: Why do we still see corruptions?

Clearly the above is not enough. Some of the interesting cases are

· What if any of the underlying systems ‘SD card software’ or FAT/TFAT has a bug?

· Suspend/resume cycle can result in file buffer loss if SD card is used. So the changes supposed to be written to disk could be lost.

The intent is not to blame other components. However in the last 3 years, at least 90% of the corruption bugs reported are due to bad checksum. It means disk write is bad for some reason (which is not owned by Compact). The fault is somewhere in the bottom stack and it has to be fixed. Our options are very limited without a repro but we are trying hard to improve our debugging infrastructure. We are hoping that with this knowledge customers will be able to understand the corruptions better and come up with good practices that would vastly reduce these corruptions.

Thanks

Raja

"Houston, we got some problem..." and "Houston, we got a problem..." - notice a difference?

SQLCEBLOG — Wed, 12 Dec 2007 13:18:00 GMT

So you have run into a problem - how to efficiently report this:

System configuration:

On a Desktop/Server, Run Winver from Start -> Run.
On a Pocket PC, Start -> Settings -> About
On a Smart Phone, a bit more steps - start -> Settings -> More ->About

Give us all the details you see here.

SQL CE Version:

Applies if you had installed SQL CE by various means <VS / SSMS etc.>
Search for the sqlcese*.dll, right click it and give us the version in version tab.

A detailed description of what you are trying to do:

Include all details like whether you are using Managed /Native code, single/multi-process and a sample code
A small repro of the error you are getting would be great! <In case of corrupt databases, the corrupt database>

A Dump file if the process has crashed / hung:

On a PC, install this wonderful tool - UserDump.exe from http://support.microsoft.com/kb/241215. This tool installs a control panel entry - click this and register your application's name on this. Next time you get a crash in that app, you will get a dump file in the location you specified. Neat isn't it?

In case of a hang, use the same tool above to assign a hotkey and dump the process's current state.

A word of caution, this crash dump file can be pretty big - so ensure you have enough disk space.

On a mobile device, this gets a bit trickier. If you are using Windows CE 5.0+ version, and if you are lucky enough to have an OEM vendor who has enabled Windows Error Reporting, you can get a dump. Read this wonderful article at http://windowsfordevices.com/articles/AT7568439504.html by Abraham Kcholi and Gad Meir.

For older devices <Win CE 4.2 and so on>, contact us with the program and the problem, and we will handle it case by case.

P.S: No, don't upload all the attachments on to the forums:), but keep them handy. The moment we start looking at this, we will be asking for these.

P.S.2: First post here - w00t! I am a developer in the engine team of SQL CE and I guess you will see me here time and again!

Nandri!

Subu Subramanian.

There can be hope only for a society which acts as one big family, not as many separate ones. - Anwar el Sadat

SqlCeEngine/ISSCEEngine: Shrink Vs Compact

SQLCEBLOG — Fri, 26 Oct 2007 05:42:00 GMT

The difference between these two is much similar to Internal and External Memory Fragmentation.

From SqlCeEngine.Shrink documentation,

Reclaims wasted space in the database by moving empty and unallocated pages to the end of the file, and then truncating the file. You can configure a database to automatically shrink by setting the autoshrink threshold option in connection string. Shrink does not create a temporary database file.

From SqlCeEngine.Compact documentation,

Reclaims wasted space in the database by creating a new database file from the existing file. By creating a new database means, it reclaims the free space between rows.

To be more clear, Shrink claims the pages which are entirely free or unallocated; where as, Compact claims the wasted space with in the page too. Hence Compact requires creating a new database file.

Empty space with in the page could be as result of:

1) If there were 5 rows in a page and two of them are deleted

2) If there was a row in the middle of the page which on update required more space, has moved out of the page (Ex: nvarchar column update)

3) Etcetera

Empty pages and unallocated pages could remain in database as a result of:

1) All rows in a page are deleted

2) Whole table is dropped

3) Etcetera

Thanks,

Laxmi Narsimha Rao ORUGANTI

Connection String - Database Creation-time Only Options

SQLCEBLOG — Wed, 13 Jun 2007 22:47:00 GMT

In this article, we will discuss the Connection String Parameters that are meant for use only for database creation and not for connection open to an existing database.

Locale Identifier – LCID tells the engine in which locale/collation the database should be created.

Encrypt – As the name suggest is to encrypt a database. It is to tell engine to create an encrypted database. This is going to be deprecated from v3.5 onwards. Please use Encryption Mode instead.

Encryption Mode – This is a new connection string parameter for v3.5. It has three values: Platform Default (DEFAULT), Engine Default, and PPC 2003 Compatibility. This option tells the engine to create a database of platform compatibility to support for the encryption. Platform Default means try Engine Default if possible, otherwise go with PPC 2003 Compatibility. Engine Default tells the engine to use the algorithms that may NOT be compatible across all platforms (Which means a DB created on Windows CE 5.0 platform cannot be opened on Windows CE 4.2 platform), on the other side PPC 2003 Compatibility tells the engine to use algorithms such a way that DB file created on Window CE 5.0 platform can be opened on any SQL CE supported platform.

Note that these connection string creation-time only options are used while compacting a database too, as in case of compact also we create a fresh destination database and may over write the source database depending on the options chosen for compact.

Thanks,

Laxmi Narsimha Rao ORUGANTI

The Story of Max Database Size (Connection String Parameter)

SQLCEBLOG — Wed, 13 Jun 2007 22:28:00 GMT

Maximum Database Size is NOT a persistent configuration (It is NOT stored in database file). SQL CE creates shared memory to support multi-connection scenarios. SQL CE maintains an entry for each page of the database file, in this shared memory. Because we don't always grow to 4 GB, it is not wise to allocate shared memory to accommodate all page entries/references to support 4 GB. This means, we should have a default for max no. of page references (max database size) in shared memory to be reserved.

Maximum Database Size is defaulted to 128 MB on devices and 256 MB on desktops (for v3.1).

But sometimes user knows that it may grow beyond the default Maximum Database Size and hence should be provided with a configuration parameter in connection string. As you can see this connection string parameter Max Database Size is helpful only for the connection that is creating the shared memory and reserving the page reference part; which is obviously the first connection to database. (Note: For every database file, there exists only one shared memory). The 2nd connection cannot do anything with this connection string parameter as there is exists a shared memory for that database, 2nd connection has to just rely upon the existing configuration. The moment all connections are down, the shared memory is deleted. That is why you can reconfigure the Max Database Size for every first connection to the database file.

The story holds good for other connection string parameters like ‘Max Buffer Size’.

Thanks,

Laxmi Narsimha Rao ORUGANTI