Welcome to MSDN Blogs Sign in | Join | Help

Troubleshooting: Access Violation exception while using SQL Server Compact database with ADO.NET provider

 

Possible causes for Access Violation exception in general:

 

An access violation occurs in unmanaged or unsafe code when the code attempts to read or write to memory that has not been allocated, or to which it does not have access. This usually occurs because a pointer has a bad value.

In programs consisting entirely of verifiable managed code, all references are either valid or null, and access violations are impossible. An AccessViolationException occurs only when verifiable managed code interacts with unmanaged code or with unsafe managed code.

 

For more information about this, please visit this link.

 

When this exception can be related to SQL Server Compact:

 

SQL Server Compact ADO.NET provider DLL [System.Data.SqlServerCe.dll] is a managed DLL which interacts with other native DLLs to provide the functionality.

 

There are two known issues related to the usage of SQL Server Compact provider which may lead to this Access Violation exception:

 

1.      Version Mismatch [3.5 RTM and 3.5 SP1]:

 

Access Violation occurs when the loader loads different versions of Managed and Native SQL Server Compact 3.5 DLLs. This issue arises when different versions are available in the box and this happens with Private and Central deployment of SQL Server Compact 3.5.

 

To be specific, if SQL Server Compact 3.5 RTM is pre-installed [centrally deployed] in the box and application built on top of SQL Server Compact 3.5 SP1 privately deploys the SQL Server Compact DLLs.

 

In this case, the managed DLL [System.Data.SqlServerCe.dll v3.5.5692.0] will be loaded from private directory but the native DLLs [sqlceme35.dll, sqlceqp35.dll, etc. v3.5.5386.0] might get loaded from the central deployed directory.

 

If the application is built against SQL Server Compact 3.5 RTM and SQL Server Compact 3.5 SP1 is centrally deployed in the box then the application will start using SQL Compact 3.5 SP1 as there is a redirection entry which redirects an application (Compiled against 3.5 RTM) to use centrally deployed SQL Compact 3.5 SP1.

   

2.      Thread Safety:

 

SQL CE objects are not thread-safe and are not thread affinitive either. If an instance of SqlCeConnection or SqlCeTransaction is shared across threads without ensuring thread safety, then that may lead to Access Violation exception.

 

 

Work-around if the cause is related to SQL Server Compact:

 

For the above two issues, there are workarounds available which should be followed to avoid any Access Violation exception.

 

1.      Version Mismatch:

 

One should follow either of the given below approaches to avoid the Access Violation exception:

 

Scenario: Application built against SQL Server Compact 3.5 SP1 [to be deployed privately] and SQL Server Compact 3.5 RTM centrally deployed.

 

Solution:

1.  There should be only one version of SQL Server Compact [v3.5 RTM or v3.5 SP1] installed in the box. The version with which the application is not built against [v3.5 RTM] should be un-installed.

2.  Install SQL Server Compact [v3.5 SP1] centrally to avoid any future install mismatch.

 

Forum thread related to this issue can be found here.

 

2.      Thread Safety:

 

It is recommended that each thread should use a separate connection than sharing across. If there is really a need for sharing SQL CE objects across threads, then the application should serialize access to these objects.

 

Thanks,

Mohammad Imran.


SQL Server Compact Garbage Collection – Whys and Hows

Some background about SQL Server Compact:

SQL Server Compact is an embedded database implemented in native and it can be accessed in application by either OLEDB or ADO.NET provider model.

ADO.NET provider for SQL Server Compact is a managed assembly which depends on native SQL Server Compact DLLs to provide the service. In general, the managed classes under ADO.NET provider model for SQL Server Compact are just wrappers around the native classes.

SQL Server Compact native objects are dependent. For instance, Cursors and Transactions are dependent on Connection object and are expected to be released in the right order.

What’s the problem then? In native application, the developer can dispose the cursors, commands, etc. before closing the connection but in managed application with .Net Garbage Collector there is no particular order in which these objects will be disposed.

So what do we do? Our SqlCeConnection object maintains weak references to all objects tied with it. Or to be specific, it maintains short weak references.

How does it help me in disposing the objects in order? When the connection object is getting disposed, the weak reference cache is iterated and all the objects there are disposed before the connection object getting disposed. Please note that when we say Dispose, we also mean Finalization.

How does .Net Garbage Collector comes into picture of SQL Server Compact

Simply employing short weak references to keep track of all related objects and dispose them in sequence does not solve all the problems.

What is the problem then? Sometimes the database files can remain locked in stress scenarios because not all the dependent objects are disposed upon dispose of SqlCeConnection. This happens because we use short weak refs to track object lifetime and we could end up in situations in which an object is in the finalization queue (hence the short reference is not longer alive) and isn’t in the freachable queue (hence isn’t guaranteed to be picked up by the GC immediately). More information about short weak references can be found here.

All this leads to a situation, where the database file is still locked even though customer application closed the connection.  Hence, it is not able to get deleted the file using file explorer …etc means.

How did we solve the problem? When we are cleaning weak reference cache in dispose of  SqlCeConnection object, we need to call GC.WaitForPendingFinalizers()  before we return to the caller; in this way, we are guranteed that the  GC will pick up all finalizable objects and hence all the native  interfaces will be properly released even if we no longer have short  references to them.

More information about this API can be found here.

FAQs

1. Why are we employing short weak references and not long weak references?

Ans.: SQL Server Compact ships for both Desktops as well as Devices. Long weak references are not supported in .Net CF. More information here.

2. Does call to GC.WaitForPendingFinalizers() has negative performance impact?

Ans.: No. Most of the time, the freachable queue  is empty (the same is not true for finalization queue though). See here for more information about Garbage Collector.

3. Can this call to GC.WaitForPendingFinalizers() lead to deadlocks?

Ans.: Yes. If Dispose method of SQL CE objects are called in the finalization context than the explicit dispose context. The reason being when an object is getting finalized, it is not supposed to touch any other managed objects as the currently getting disposed object does not know the life-status of the object it is trying to refer.

4. Why did we choose this design?

Ans.: SQL CE was primarily designed for Devices where the memory, processing comes at very high cost.  So, it has been designed to free up the resources as early as we can and hence it is calling GC.WaitForPendingFinalizers.

Thanks,

Mohammad Imran.

After moving the database from one platform to other, the first SqlCeConnection.Open() takes more time

 

 

If you move a SQLCE database from one platform to other, it's first SqlCeConnection.Open() takes more time and also increases the database file size.

 

The Reason: For an index on string type columns, SQLCE uses LCMapString API to get the normalized sort key. LCMapString API behavior will be different for different NLS sort versions. NLS sort version is different for Desktop Windows OS and WM/WCE. During the first database connection open, SQLCE rebuilds all indexes if there is a NLS sort version mismatch with the last accessed OS platform. Index recreation requires space on database file and hence the database file size will be increased. You can use Compact()/Shrink() API to get the database file with approximately old file size.  Because of this index recreation, first database connection open takes more time as it has to rebuild all indexes.  

Note: If your database doesn’t have any indexes you will not see above mentioned symptoms on first connection open.

 

 

Don’t inter-op SQLCE 3.0 databases between devices and desktop:

SQLCE started supporting desktops only from SQLCE 3.1 (3.0.5300.0) release. SQLCE 3.0 doesn’t have index rebuilding logic and the index re-creation was added during SQLCE 3.1 (3.0.5300.0) release as it is required to support platform inter-op for SQLCE databases.  On desktop you can open SQLCE 3.0 databases, which were created on devices. Again If you open these databases on SQLCE 3.0 devices, your device app might receive an error/exception.

 

For example: Opening a SQLCE 3.0 database with SSMS will use SQLCE 3.1 bits on desktop and hence it would drop and recreate index data. When this file is copied back to device, index data may not be valid and receive an error/exception.

 

Namaskaram!

Manikyam Bavandla

 

Overview of SqlCeReplication methods - LoadProperties and SaveProperties

Whenever sync is started, developers provide all the necessary properties to an object of SqlCeReplication class. These properties are provided through the constructors or by directly setting the public properties of the object. In this approach developers need to provide this data for every sync, which forces them to store this data somewhere in their code. Even if the properties seldom change, developers provide this data all the time. This data includes passwords such as DistributorPassword, InternetPassword, InternetProxyPassword and PublisherPassword which developers are better off, not saving or handling in their code.

This blog article throws light on SaveProperties/LoadProperties methods which provide a good solution to issues described above. SaveProperties saves all the properties a developer can provide to SqlCeReplication object (except for two which are described below) in a system table called __sysMergeSubscriptionProperties within the replicated database. The benefits of this approach are:

1)      The replicated database can be encrypted securing this data inside the sdf file.

2)      Passwords are encrypted with the unique device hardware key enabling an extra layer of security.

3)      Data that seldom changes need not be explicitly provided by developer for every sync

Code example for calling SaveProperties:

// Example code for SaveProperties starts

repl = new SqlCeReplication();

repl.InternetUrl = "http://www.adventure-works.com/sqlmobile/sqlcesa35.dll";

repl.InternetLogin = "MyInternetLogin";

repl.InternetPassword = "<enterStrongPassword>";

repl.Publisher = "MyPublisher";

repl.PublisherDatabase = "MyPublisherDatabase";

repl.PublisherLogin = "MyPublisherLogin";

repl.PublisherPassword = "<enterStrongPassword>";

repl.Publication = "MyPublication";

repl.Subscriber = "MySubscriber";

repl.SubscriberConnectionString = "Data Source=MyDatabase.sdf";

 

repl.SaveProperties();

 

// Example code for SaveProperties ends

Note: SaveProperties will succeed only if the Subscription is already created on the machine.

 

This simplifies subsequent syncs in that developers need to provide only SubscriberConnectionString (to identify the local database) to SqlCeReplication object and call LoadProperties. If __sysMergeSubscriptionProperties table has only one record this data is automatically loaded into SqlCeReplication object. If there are more records, then in additional to SubscriberConnectionString, developers need to provide Publication, Publisher and PublisherDatabase to uniquely identify a saved profile for a subscription.

Code example for calling LoadProperties:

// Example code for LoadProperties starts

repl = new SqlCeReplication();

repl.SubscriberConnectionString = "Data Source='Test.sdf'; Pwd='<enterStrongPassword>'";

 

repl.LoadProperties(); // assuming only one subscription’s properties are saved

 

// Example code for LoadProperties ends

LoadProperties/SaveProperties thus provide an efficient way to store and retrieve data relieving developers from maintaining sensitive and unchanging data.

Properties not saved by SaveProperties:

SendTimeout and ReceiveTimeout properties are not saved via SaveProperties. These values must be explicitly provided to SqlCeReplication object for every sync.

Note:

There are two things to note while using this functionality.

1)      Before calling SaveProperties, please make sure all necessary properties are set on the SqlCeReplication object. If we call SaveProperties with required properties missing, this data will be set to default even if there is a saved profile already present for this subscription in the Properties table.

2)      Passwords stored are encrypted with the unique device hardware key. Therefore if the database file is moved to a different machine LoadProperties will fail to retrieve the correct password. Developers need to provide the password and other required properties, and call SaveProperties again to refresh the saved profile on the new machine.

For more information please refer MSDN documentation:

a)      SqlCeReplication.SaveProperties - http://msdn.microsoft.com/en-us/library/system.data.sqlserverce.sqlcereplication.saveproperties.aspx

b)      b) SqlCeReplication.LoadProperties - http://msdn.microsoft.com/en-us/library/system.data.sqlserverce.sqlcereplication.loadproperties.aspx

Thanks,
Balasubramaniam Bodedhala

 

Posted by SQLCEBLOG | 1 Comments
Filed under:

How to: Authoring a 64-bit ClickOnce Bootstrapper package in Visual Studio 2008 SP1

 

Before going into further details first “Disclaimer: This is just to help and does not guaranty that this is approved my employer or me J

 

Location of SQL Compact Bootstrapper

        <BootstrapperDir>\Packages\SQL Server Compact Edition\<LocaleCultureDir>\

Where,

·         <BootstrapperDir>  can be found by reading [HKLM\Software\Microsoft\GenericBootstrapper\3.5]path value otherwise its default value is %ProgramFiles%\Microsoft SDKs\Windows\v6.0A\Bootstrapper\ .

·         <LocaleCultureDir> is culture specific dir, ex: EN, DE, zh-CHS, zh-CHT, …

 

 

Follow the below steps, If you have an ‘Any CPU’ ClickOnce application using SQL Compact 3.5 SP1, for which you want to install SQL Compact x64 MSI on 64-bit machine.

Note: It is required to install both x86 and x64 MSIs on 64-bit OS, since x86 MSI is pre-requirement for x64 MSI.

 

Steps:

1)      Download the SQL Compact 3.5 SP1 x64 MSI and place it under <BootstrapperDir>\Packages\SQL Server Compact Edition\<LocaleCultureDir>\

 

2)      ClickOnce setup.exe is always runs as WoW process on x64 OS, that’s why reading a registry key is not useful for x64 MSI. We can achieve the install check using  <MsiProductCheck> element.

 

Edit the <BootstrapperDir>\Packages\SQL Server Compact Edition\<LocaleCultureDir>\Package.xml file with following details

·         As part  of <InstallChecks> element add <MsiProductCheck> as below

<InstallChecks>

    ---

  <MsiProductCheck Property="SQLCompactRunTimex64Installed" Product="[ProductCode]" />

    ---

</InstallChecks>

 

[ProductCode] is the Product Code/GUID of the x64 MSI. Depending on locale, get it from below table for 3.5 SP1 x64 release.

You can use Orca.exe to get the ProductCode property of a MSI.

 

Locale

Three letter locale Id

Product Code

Chinese

CHS

{8DD60183-76ED-4416-8C9C-E5A39E1826EF}

Chinese (Taiwan)

CHT

{A423B3FB-C9E6-4953-9A83-2A5F45CAF466}

German

DEU

{77CB2F9F-67C5-4ADA-9321-B30C9C64727E}

English

ENU

{F83779DF-E1F5-43A2-A7BE-732F856FADB7}

Spanish

ESN

{5B32AC72-6251-47F4-BD1B-AD479E3EEBA9}

French

FRA

{A64CF374-A3DA-4B1E-A42A-6394C48F431A}

Italian

ITA

{4634B103-984E-4F31-BD80-DCD83AEEEF85}

Japanese

JPN

{1A22CAF6-E6FD-4D65-AEBA-F28D23B68EBF}

Korean

KOR

{38514244-6C25-42EC-B144-276F6DDAC9CE}

Portuguese (Brazil)

PTB

{C8445DBB-783F-4804-B373-D5CDC0614E60}

Russian

RUS

{CAA59A81-E35A-4582-80FF-F19520FFF60F}

 

3)      Add <PackageFile> element for x64 MSI. Replace [ThreeLetterLocaleID] with three letter locale id mentioned in above table.

 

<PackageFiles>

    ---

<PackageFile Name="SSCERuntime-[ThreeLetterLocaleID]-x64.msi"    PublicKey="3082010a0282010100a2db0a8dcfc2c1499bcdaa3a34ad23596bdb6cbe2122b794c8eaaebfc6d526c232118bbcda5d2cfb36561e152bae8f0ddd14a36e284c7f163f41ac8d40b146880dd98194ad9706d05744765ceaf1fc0ee27f74a333cb74e5efe361a17e03b745ffd53e12d5b0ca5e0dd07bf2b7130dfc606a2885758cb7adbc85e817b490bef516b6625ded11df3aee215b8baf8073c345e3958977609be7ad77c1378d33142f13db62c9ae1aa94f9867add420393071e08d6746e2c61cf40d5074412fe805246a216b49b092c4b239c742a56d5c184aab8fd78e833e780a47d8a4b28423c3e2f27b66b14a74bd26414b9c6114604e30c882f3d00b707cee554d77d2085576810203010001" />

</PackageFiles>

 

4)      Add <Command> elements for x64 MSI.

 

     </Commands>

            <!--  Install case for the x64 redist   -->

       <Command PackageFile="SSCERuntime-[ThreeLetterLocaleID]-x64.msi" Arguments="">

<InstallConditions>

            <!--  ByPass if we have installed the x64 redist   -->

  <BypassIf Property="SQLCompactRunTimex64Installed" Compare="ValueGreaterThan" Value="0" />

            <!--  Install only on AMD64 Processor   -->

  <BypassIf Property="ProcessorArchitecture" Compare="ValueNotEqualTo" Value="AMD64" />

            <FailIf Property="AdminUser" Compare="ValueEqualTo" Value="false" String="AdminRequired" />

  </InstallConditions>

   <ExitCodes>

  <ExitCode Value="0" Result="Success" />

  <ExitCode Value="1641" Result="SuccessReboot" />

  <ExitCode Value="3010" Result="SuccessReboot" />

  <ExitCode Value="4123" Result="SuccessReboot" />

  <DefaultExitCode Result="Fail" String="Anunexpected" FormatMessageFromSystem="true" />

</ExitCodes>

       </Command>

 

            <!--  Reinstall/Repair case for the x64 redist   -->

       <Command PackageFile="SSCERuntime-[ThreeLetterLocaleID]-x64.msi" Arguments="REINSTALL=ALL">

<InstallConditions>

            <!--  Check if we haven’t  installed the x64 redist, no need to repair it   -->

  <InstallIf Property="SQLCompactRunTimex64Installed" Compare="ValueGreaterThan" Value="0" />

            <!--  Install only on AMD64 Processor   -->

  <BypassIf Property="ProcessorArchitecture" Compare="ValueNotEqualTo" Value="AMD64" />

            <!--  This is the actual condition for reinstalling the x64 MSI  -->

  <BypassIf Property="ENU_INST_GAC" Compare="ValueExists" />

  <FailIf Property="AdminUser" Compare="ValueEqualTo" Value="false" String="AdminRequired" />

</InstallConditions>

<ExitCodes>

  <ExitCode Value="0" Result="Success" />

  <ExitCode Value="1641" Result="SuccessReboot" />

  <ExitCode Value="3010" Result="SuccessReboot" />

  <ExitCode Value="4123" Result="SuccessReboot" />

  <DefaultExitCode Result="Fail" String="Anunexpected" FormatMessageFromSystem="true" />

</ExitCodes>

       </Command>

      </Commands>

5      

          5) For making AnyCPU SQL Compact 3.5 SP1 Clickonce app to download 64-bit and 32-bit MSIs from microsoft download site (In this case, you can skip step 1), you need to do following additial things.

              a) Add HomeSite attribute to the 64-bit <PackageFile> element. 

<PackageFiles>

    ---

<PackageFile Name="SSCERuntime-[ThreeLetterLocaleID]-x64.msi" 

  HomeSite="HomeSiteName_64"   

  PublicKey="3082010a0282010100a2db0a8dcfc2c1499bcdaa3a34ad23596bdb6cbe2122b794c8eaaebfc6d526c232118bbcda5d2cfb36561e152bae8f0ddd14a36e284c7f163f41ac8d40b146880dd98194ad9706d05744765ceaf1fc0ee27f74a333cb74e5efe361a17e03b745ffd53e12d5b0ca5e0dd07bf2b7130dfc606a2885758cb7adbc85e817b490bef516b6625ded11df3aee215b8baf8073c345e3958977609be7ad77c1378d33142f13db62c9ae1aa94f9867add420393071e08d6746e2c61cf40d5074412fe805246a216b49b092c4b239c742a56d5c184aab8fd78e833e780a47d8a4b28423c3e2f27b66b14a74bd26414b9c6114604e30c882f3d00b707cee554d77d2085576810203010001" />

</PackageFiles>

b) Add <String> for HomeSiteName_64 in <Strings>

 <Strings>

       ---

      <String Name="HomeSiteName_64">[x64MSIDownloadURL]</String>

       ---

 </Strings>

 

           In place of [x64MSIDownloadURL] put download URL for locale specific x64 Runtime MSI.

           Below is the list of locale specific download URLs for x64 3.5 SP1 Runtime MSIs. 

English: http://download.microsoft.com/download/8/4/2/8423C019-CCB4-4D7D-B7F0-BCF83F1B9218/SSCERuntime-ENU-x64.msi

German: http://download.microsoft.com/download/1/5/1/1517BA39-EAC3-4281-900B-AFB77369169A/SSCERuntime-DEU-x64.msi

Spanish: http://download.microsoft.com/download/C/B/E/CBE7CA6E-0D13-4186-A3CD-03C74CE9A0FA/SSCERuntime-ESN-x64.msi

French: http://download.microsoft.com/download/3/A/3/3A3B694E-9742-4AFF-8357-DCC9BAAE8A40/SSCERuntime-FRA-x64.msi

Italian: http://download.microsoft.com/download/9/E/F/9EF6D714-80EA-4373-962C-60436ED8A1D8/SSCERuntime-ITA-x64.msi

Japanese: http://download.microsoft.com/download/D/7/1/D717C79C-D847-42BF-BB9C-C1C390D203C4/SSCERuntime-JPN-x64.msi

Korean: http://download.microsoft.com/download/B/5/7/B5773FB4-1593-4746-824D-15235D820EFC/SSCERuntime-KOR-x64.msi

Portuguese (Brazil): http://download.microsoft.com/download/D/3/8/D388E5C0-8671-4F6A-8A05-86B9470B4E97/SSCERuntime-PTB-x64.msi

Russian: http://download.microsoft.com/download/2/3/E/23EB9CB1-F037-463F-BE83-AD3848FD1480/SSCERuntime-RUS-x64.msi

Chinese: http://download.microsoft.com/download/9/A/0/9A04CFE5-29CB-4B5C-A5F9-C66CB5FD1A9B/SSCERuntime-CHS-x64.msi

Chinese (Taiwan): http://download.microsoft.com/download/D/E/B/DEB711BD-FE6A-45E9-AA15-D9901E8458C3/SSCERuntime-CHT-x64.msi

 

 

 

ClickOnce Reference: http://msdn.microsoft.com/en-us/library/ms229223(VS.80).aspx

 

 

Thanks

Manikyam Bavandla

 

Posted by SQLCEBLOG | 1 Comments

Sync Services optimizations


    There are some optimizations that can be done for an "Sync Services" sync scenario. These are, (might not be an exhaustive list)

  1. Small sync chunks (using smaller SyncGroups which are only absolutely necessary)
  2. Index creation on tracking and filtering columns
  3. Transfer encryption of data payload on web
  4. Batching the changes                 
  5. Filtering the data at various stages to reduce network load

    Sync (synonym for "Sync Services sync", in this post) interfaces with the providers using DataSet objects. That is, sync uses DataSet to store and retrieve changes happening at client and server. DataSets are known to be not very memory efficient, and the amount of memory consumed by these, is a bit of a concern when one is using these in device clients in Sync Services. 
 
    Sync happens in chunks, using the SyncGroups that one supplies. That is, all data in a SyncGroup is logically one sync, and corresponds to one DataSet. Since sync DataSets are one per SyncGroup, and these are needed only till the corresponding SyncGroup commits, we can optimize the memory taken by DataSets, using smaller SyncGroups. Divide sync tables into various SyncGroups, so that, each group has it’s own (smaller) DataSet, and it is disposed (by garbage collector) once the SyncGroup changes are committed. However, to ensure logical consistency, you should always keep the tables which are related (by a foreign key constraint, for example), in the same SyncGroup.
 
Example:
Tables Order and OrderDetails are related.
Tables Customers and ShippingDetails are related. But, these are unrelated to the first set.
 
SyncGroup customerSyncGroup = new SyncGroup("Customers");
 
SyncTable customerSyncTable = new SyncTable("Customer");
customerSyncTable.CreationOption = TableCreationOption.DropExistingOrCreateNewTable;
customerSyncTable.SyncDirection = SyncDirection.DownloadOnly;
customerSyncTable.SyncGroup = customerSyncGroup;
 
SyncTable orderShipSyncTable = new SyncTable("ShippingDetails");
orderShipSyncTable.CreationOption = TableCreationOption.DropExistingOrCreateNewTable;
orderShipSyncTable.SyncDirection = SyncDirection.DownloadOnly;
orderShipSyncTable.SyncGroup = customerSyncGroup;
 
SyncGroup orderSyncGroup = new SyncGroup("Orders");
 
SyncTable orderSyncTable = new SyncTable("Order");
orderSyncTable.CreationOption = TableCreationOption.DropExistingOrCreateNewTable;
orderSyncTable.SyncDirection = SyncDirection.DownloadOnly;
orderSyncTable.SyncGroup = orderSyncGroup;
 
SyncTable orderDetailSyncTable = new SyncTable("OrderDetails");
orderDetailSyncTable.CreationOption = TableCreationOption.DropExistingOrCreateNewTable;
orderDetailSyncTable.SyncDirection = SyncDirection.DownloadOnly;
orderDetailSyncTable.SyncGroup = orderSyncGroup;
 
 
    Rather than having a single SyncGroup with all 4 tables, one can create two sync groups. One for the first set, and the second for the other set. 
 
    Also, to increase the performance of enumerating process at server, one can create indexes on the tracking and filtering columns. As an example, let us say one has two tracking timestamp columns __CreateTime and __UpdateTime. Since, our server side changes enumerating queries always have filtering based on these columns, one should create indexes on these, so that, enumerating server changes becomes faster, and, sync performance increases. 
 
    In case of device syncs, transferring the data on the network/internet is also going to cost more time/resources. Since, transfer of data from a web service/server machine to a device is time/bandwidth consuming, one should look at compressing data sent/received. This is mostly the case, when data being synched is large (Something like, initial synching SQL Compact with SQL Server). One can use System.IO.Compression or your own implementation of compression/decompression algorithm. (More details at http://msdn.microsoft.com/en-us/library/system.io.compression.aspx). Another blog post that refers to this issue and proposes solutions is at, (http://blogs.msdn.com/mahjayar/archive/2008/10/01/dbsyncprovider-improving-memory-performance-in-wcf-based-synchronization.aspx)
 
    Sync Services Server side supports batching. Please refer to http://msdn.microsoft.com/en-us/library/bb902828.aspx for more details.  
 
    Sometimes, we may have server tables holding and synching data, that is alien to clients. For example, server table has a large GPS map of some place (in custom format) per row, that is associated with the data in other columns in the row. The map is stored as an image at client side. But, since, the data is of proprietary format, client can’t interpret the data, and it is not used anyway at client side. And also, since, it is large data, lot of network bandwidth is consumed and response time is compromised for transferring this across. It is better, if one can filter these columns out at server side, rather than sending to client. 
 
    Another scenario is as follows: The client devices of a Sync application are used by Salesmen of a company, and the server side is the company office, where data pertaining to each salesman is stored in a SQL Server database. When the client devices sync, they are mostly concerned about the data relevant to them alone, and not to other sales persons. Here also, one needs to filter rows from the table. 
 
    To facilitate filtering of rows/columns at client/server sides, one can use a number of strategies. For filtering data sent from client (or at the client side), one can use the callback architecture of "Sync Services". For server side, since you have more control there, one can choose to strip down the DataSet, when one is sending it over the wire/use callbacks to strip off DataSet data/client can supply queries to server side adapters, which filter data.

Explaining each of these options:

    Callback architecture of sync: Sync provides various callbacks during the process of Synching. One can hook in the callbacks for accessing/modifying the dataset generated at every stage of sync. For example, one can add/drop some columns to the dataset, after changee enumeration is done at client side. After changing the dataset, effectively, the changes being applied to client database are changed. The same provision is present at server side also.

    This callback architecture can be used to modify the DataSet applied/sent to server, on the fly. This is one type of filtering, possible at both client and server.

    Server side gives more options when sending data over the wire. First of all, server chnages are enumerated within the "Sync Services" application itself. That is because, server change enumeration is done by queries supplied by the Sync Services client. So, which columns to select from server, can be controlled from the client side, when specifying queries like, SelectIncrementalInserts etc... This is another way of programmatically filtering server data.

    One more option to filter data in a N-tier architecture is that, filtering the wire transferred content. In an N-tier architecture, client and server are in two different, but connected machines. Typically, client machine has a proxy server provider, which delegates calls to enumeraion/application of changes etc... to a web service. Web service talks to a backend data store, and services the data and operation requests. In this setting, interesting possibilities like, use of compression over the wire, and filtering over the wire etc... come up. When server side data store gives data, it can do some filtering. Also, the client side proxy can be used to filter irrelevant data. This is another type of filtering.

Example for callback architecture:
For client side DataSet filtering (client side), use SelectingChanges before the enumeration and ChangesSelected after the enumeration
For server’s data set filtering (client side), ApplyingChanges before the application of changes and ChangesApplied after the application
For filtering server side changes (at server side):SelectingChanges before the enumeration and ChangesSelected after the enumeration
For client’s data filtering (at server side):ApplyingChanges before the application of changes and ChangesApplied after the application
 
In every one of these methods, eventArgs.GroupMetadata and eventArgs.Context.DataSet are the ones need to be changed.

 Thanks

Udaya Bhanu,

SDE II, SQL Compact

Posted by SQLCEBLOG | 0 Comments

Released - SQL Server Compact v3.5 SP1, SQL Server 2008 RTM, Visual Studio 2008 SP1, .NET Framework v3.5 SP1

Yes, all of them are released on August 11, 2008. 

SQL Server Compact v3.5 SP1:

What is new in this Service Pack? - Click here and here

Where to download?

SQL Server Compact v3.5 SP1 for Desktops (32-bit and 64-bit)

SQL Server Compact v3.5 SP1 for Devices (all platforms & processors)

Server Tools (32-bit and 64-bit) (See Also: this)

Documentation, Books Online

Visual Studio 2008 SP1

What is new in this Service Pack? - Click here

Where to download? - Click here

 

.NET Framework v3.5 SP1 - Download it here

SQL Server 2008 RTM - Click here

The good thing is with VS 2008 SP1, you will get "SQL Server Compact SP1 for Desktop (32-bit)", and of course updated "SQL Server Compact SP1 Design Tools".

We received a huge feedback from you on VS 2005 SP1 that we should limit the Service Pack Size.  With that feedback, we felt it is a reasonable to ask customers to download "SQL Server Compact v3. SP1 for Devices" explicitly than including it with Visual Studio 2008 SP1 there by bloating VS 2008 SP1 size.  We also made a decision to not include "SQL Server Compact v3.5 SP1 for Desktops (64-bit)" so as to make sure we do not trouble all VS 2008 SP1 customers.

What about Platform Builder Updates?

Platform Builder 5.0 and 6.0 Updates for SQL Server Compact v3.5 SP1 are also released to Platform Builder team and they should be available in the next Platform Builder 5.0 and 6.0 Update.  I will update this blog post with the download URL soon after they are available. 

Windows CE 5.0 Platform Builder Monthly Update (July 2008) - Download it here

Windows Embedded CE 6.0 Platform Builder Monthly Update (July 2008) - Download it here

 

Thanks,

Laxmi Narsimha Rao Oruganti

Posted by laxminro | 1 Comments
Filed under:

Insight into SQL Server Compact database corruptions

Database corruptions are often the most painful bugs to repro and debug. A good percentage of the SQL Server Compact bugs reported are in this category. However it so turns out that in majority of the cases Compact is not the culprit. Surprised!! Let us see why.

How SQL Server Compact detects a file corruption?

Compact database comprises of a series of pages, each 4K in size. When a page is written to disk the following sequence of steps is executed.

 

1.       Compute the checksum on the page data and store it as part of the page.

2.       Encrypt the page.

3.       Write to disk at a given file location or offset.

4.       Update system pages/run time cache with the page offset.

 

When a page is read from disk the following sequence of steps is executed.

 

1.       Read from the disk.

2.       Decrypt the page.

3.       Re-compute the checksum and compare it with the expected value. (If it fails page is not fully baked. Meaning file write failed)

4.       Check if this is the same page that we are expecting as per the system pages/ run time cache. (If this fails we lost a complete page write)

 

If (3) or (4) of the above read steps fail, the db is marked as corrupt and any further db activity is suspended till it is repaired successfully.

 

Notes:

·         Verify reports step (3) errors as “Bad Checksum - idPage: %d, tyPage: %d, iPage: %d”.

·         Verify reports step (4) errors as “Page with valid ordinal not found - idPage: %d, iPage: %d”

What could have triggered the corruption?

1.       A simple device on/off or suspend/resume can result in half-baked file. (No reason to panic as we handle this as described in the next section.)

2.       Compact relies on underlying FAT/TFAT file system to manage the database file. Any bugs in these components can lead to Compact file corruption. Compact stresses file system in many different ways as it tries to keep the file size as small as possible. As it employs shadow paging the writes and reads are quite random. (Note: 6 months back a bug has been identified in FAT/TFAT system that results in a Compact file losing some of the clusters during shrink operation.)

3.       If SD card is being used for db file storage, it’s being equally stressed as the above.

4.       More importantly SD cards bring in additional complexities during Suspend/Resume cycles. Here is a good blog link: http://blogs.msdn.com/windowsmobile/archive/2007/01/12/everything-you-want-to-know-about-sd.aspx

Am I suggesting SQL Server Compact bugs can’t result in any kind of corruption?

Of course Compact code bugs can result in corruption. A lot of them have been found internally before v3.0 release and also reported by customers thereafter. They are more logical in nature like index constraint is compromised or a table page says there are 1000 rows when there are only 55.  We did a very good job in fixing these and they are hard to find now. However Compact bugs don’t result in physical corruptions like “checksum errors” or “page missing” errors.

What does Compact do to prevent corruptions?

We believe most of the corruptions should be due to device suspend/resume or on/off cycles.  For ex: Compact is trying to write to a file and the user simply boots his device. Now file is in half-baked state. Or let’s say process gets killed before it can complete the flush cycle. How do we avoid this?

 

Well, we use shadow paging to overcome this situation. We can loosely categorize the db pages into user data pages, system pages and header page. As the name suggests user data pages contain all the user data. Header pages and system pages help us locate a specific user page on the disk. When it’s time to flush the changes to disk, we write shadows of the user data pages first. Then the shadows of the updated system pages are written. Finally the header page is written. The old pages are retained till the header page is updated. This ensures that we have a transactional write to the disk. Either the change is completely present or not.

Conclusion: Why do we still see corruptions?

Clearly the above is not enough. Some of the interesting cases are

·  What if any of the underlying systems ‘SD card software’ or FAT/TFAT has a bug?

·  Suspend/resume cycle can result in file buffer loss if SD card is used. So the changes supposed to be written to disk could be lost.

         

The intent is not to blame other components. However in the last 3 years, at least 90% of the corruption bugs reported are due to bad checksum. It means disk write is bad for some reason (which is not owned by Compact).  The fault is somewhere in the bottom stack and it has to be fixed. Our options are very limited without a repro but we are trying hard to improve our debugging infrastructure. We are hoping that with this knowledge customers will be able to understand the corruptions better and come up with good practices that would vastly reduce these corruptions. 

Thanks

Raja

Posted by SQLCEBLOG | 3 Comments
Filed under: ,

Katmai new datatypes support

In Katmai, new datatypes are introduced. Some of these (including some older types in Yukon) are supported for merge replication synching with SQL Compact in SQL Compact 3.5 SP1. These types are mapped to downlevel compatible types for SSC. These types and their mappings are, as follows:

 

SQL type                              Compact type                  Format                               

Date                                     nvarchar(10)                   ‘yyyy-mm-dd’

Time                                     nvarchar(16)                   ‘hh:mi:ss.nnnnnnn”

Datetime2                            nvarchar(27)                   ‘yyyy-mm-dd hh:mi:ss.nnnnnnn’

DatetimeOffset                    nvarchar(34)                  ‘yyyy-mm-dd hh:mi:ss.nnnnnnn [+|-]hh:mi’

Geometry                             image

Geography                           image

 

The format of strings used when inserting/updating the data on SQL Compact, should be same as that given above, or as the type demands, in case of Spatial types.

 

Spatial types are interesting, because, one can map them to image columns on the client without data loss. The data can even be manipulated on the client side, if the required spatial type libraries are present. Other new types in Katmai, like, HierarchyID etc... are not supported for merge with SQL Compact.

 

Thanks

Udaya

Posted by SQLCEBLOG | 1 Comments
Filed under:

Query performance

Understanding the reason for slowness of a SQL query and then tuning to boost it is a slightly complicated process, but it can be extremely rewarding in some cases. In this post, I am going to list down some ways in which you can tune the query performance for SQL Server Compact, along with pointers to relevant online resources. If you are a seasoned database developer, you probably know most of the tricks already.

Before I jump into it, an overview of the basics first.

Basics:

  •  Displaying actual and estimated Execution plans for SQL Server Compact: You should be familiar with Execution Plans (also called as query plans and showplans).
    • For non-parameterized queries: There are several easy ways to do this
      • When database is on a docked device or desktop: Generate a query plan using SQL Server Management Studio (SSMS) that gives a nice graphical view of the query plans. (You can connect to the database on device as long as it is docked. No need to copy the file to desktop.)
      • When database is on a device: Generate a query plan using Query Analyzer. This generates a .sqlplan file that can be viewed graphically in SQL Server Management Studio as before (preferable) or opened as a plain XML file.
    • For Parameterized (as well as non-parameterized) queries:
      • Use the TSQL Statements SET SHOWPLAN_XML  (for estimated execution plan) and  SET STATISTICS_XML (for actual execution plan), followed by SELECT @@SHOWPLAN to generate the query plans.  Store the results as an XML file. (Tip: To view the query plans graphically, just rename the file with a .sqlplan extension and it is ready to be opened in SSMS like an ordinary non-parameterized query plan.)
  • Indexes and statistics on indexes:
    • One of the important tasks of the query optimizer is to select the right index to execute the query. An optimizer can make better decisions if it has histogram data (i.e. statistics) about the distribution of values for an index.
      • CREATING/UPDATING/DELETING statistics on an index: SQL Server Compact 3.5 creates statistics on all indexes by default and updates them as required. (You can surely tweak these settings, but it is advisable not to. See CREATE STATISTICS /UPDATE STATISTICS/DROP STATISTICS).
      • To view the statistics distribution, the built in stored procedure sp_show_statistics is pretty handy. 
      • Statistics were not created by default on primary key indexes till version 3.1, so you might want to check this and add them yourself :- ).
  • ... A lot of other things too J, but this will do for the time being.
The Meat:

If you are trying to improve the performance of your application that uses SQL Server Compact, make sure you have checked the following resources on MSDN already (In this post I will not go into any of them):

  • Database Design and Performance: This page talks in some detail about the following:
    • Database denormalization
    • Variable and fixed sized columns
    • Effect of row and key lengths
  • Query Performance Tuning: This page explains some of the following:
    • Improving indexes by creating selective indexes, choosing the columns in a multi-column index, issues in indexing a small table etc.
    • When to create indexes
    • Using parameterized queries where possible.
    • Rewrite queries where possible
    • Using base table cursor (CommandType.TableDirect) mode for simple queries.

(The article on Query Process Overview and Performance Tuning approaches for SQL Server CE 2.0 covers the same points as above, but I found it better than the MSDN links J. It is relevant for SQL Server Compact 3.5 too.)

Now here are some additional ways you can improve performance:

1.       Recompile Stale query plans

<StoryTime>Sometime back I got a complaint about slow performance of our database even though the user was using parameterized queries and had created the correct indexes everywhere. The sequence in which he was performing operations was something like this:

-          Create the full database schema (tables, indexes etc).

-          Prepare SqlCeCommand objects for *all* queries that will ever be used by the application.

-          Populate the database with actual data. Until now all tables were empty.

-          Execute the queries using previously prepared command objects 

What's wrong with this?

 A prepared command object (SqlCeCommand) holds a query plan. The query optimizer generated this query plan and had optimized it for the state of database when the plan was generated. In this user's case, the database had just empty tables at that point in time. The optimizer therefore inferred that using a table scan is the best way of executing the query (as opposed to using any index). As a result, all his query plans used just table scans instead of the right indexes even when the table size increased! 

</StoryTime>

Moral of the story: Caching query plans is good, but only as long as the state of database when they are compiled is representative of the average state of database during execution. If your data is changing rapidly, it is better to recompile the queries every once in a while.

So, beware of stale query plans!

Note that query plans can be cached for both parameterized and non-parameterized queries and you can run into this problem in either case. (Another insider hack: SqlCeCommand.Prepare doesn't really prepare the query plan. It just marks the command object for plan (re)compilation and the plan is compiled when SqlCeCommand.ExecuteXXX is called next time.)

2.       Impression of using parameterized queries without really using them

Question: What's wrong (rather sub-optimal) with the following code:

SqlCeCommand cmd = con.CreateCommand();

cmd.CommandText = "SELECT * FROM table1 WHERE C1_DECIMAL = @p1";

cmd.Parameters.Add("@p1", SqlDbType.Decimal);

cmd.Parameters["@p1"].Value = 3.5;

SqlCeDataReader rdr = cmd.ExecuteReader();

while (rdr.Read()) { /* read the data */}

rdr.Close();


cmd.Parameters["@p1"].Value = 335.01;

rdr = cmd.ExecuteReader();

while (rdr.Read()) {  /* read the data */   }

rdr.Close();


Answer: The query plan gets compiled twice, once for every ExecuteReader call!

Reason: First time the plan is compiled, the parameter's precision is 2 and scale is 1 (as inferred from its value ‘3.5'). The query plan therefore uses this precision and scale values. The only other values it can accept are those that fit within this range.

When the query is executed again, the parameter's precision is 5 and scale is 2 (as inferred from its value ‘335.01'). Since it doesn't fit in the parameter as inferred during first plan compilation, the plan is recompiled silently!

Solution: If you can anticipate the range (size, precision or scale) of the parameter values, then specify it explicitly. The plan will be generated based on the specified range then. This holds true for all character, binary and numeric types. (The flip side is that for any parameter value that does not fit into the specified range, an error will be thrown.) So the above code can be modified as follows:

SqlCeCommand cmd = con.CreateCommand();

cmd.CommandText = "SELECT * FROM [TABLE1] WHERE [C1_DECIMAL] = @p1";

cmd.Parameters.Add("@p1", SqlDbType.Decimal);

cmd.Parameters["@p1"].Precision = 10;  // Playing safe by taking a larger range!

cmd.Parameters["@p1"].Scale = 5;

cmd.Parameters["@p1"].Value = 3.5;

SqlCeDataReader rdr = cmd.ExecuteReader();

while (rdr.Read()) { /* read the data */}

rdr.Close();

cmd.Parameters["@p1"].Value = 335.00;

rdr = cmd.ExecuteReader();

while (rdr.Read()){ /* read the data */ }

rdr.Close();


3.       Query optimizer did not choose the best index

The query optimizer usually does a good job of choosing the index. However, it does it on a best effort basis and there can be cases where it doesn't pick the right index. In such cases, it makes sense to use index hints and outsmart the optimizer:

E.g.  ...FROM [TABLE1] AS T1 JOIN [TABLE2] AS T2...  can be rewritten with index hints as

...FROM [TABLE1] AS T1 WITH (INDEX(IX_On_Table1))

 JOIN [TABLE2] AS T2 WITH (INDEX(IX_On_Table2))...

How do you know if your index is indeed better than the index chosen by optimizer?

Well, the easiest and brute force method is to run the query with and without an index hint and see which is better.  I often use the Query Analyzer on device and SSMS on desktop to get the execution times. Another way is to generate query plans for both queries and study them for better predicate matches, index ordering etc.

4.       Query optimizer did not choose the best join order

As in the case of choosing a wrong index, an optimizer can sometimes choose a wrong join order too. Again a corner case, but not impossible to run into. In such cases, you can specify the join order explicitly:

E.g. ...FROM [TABLE1] AS T1 INNER JOIN [TABLE2] AS T2 ON ...  can be rewritten with a forced join order as 

...FROM [TABLE1] AS T1

 INNER JOIN [TABLE2] AS T2 ON ...

 OPTION (FORCE ORDER)

 As with any sort of query hint, you should be extra careful that you are doing the right thing. For instance, the join order chosen for a query can change between multiple runs due to several reasons. The cardinality of the join tables could have changed significantly, or the data distribution could have changed. So even though the join order that you selected at the time of deployment was optimal, it may not remain so all the time. Ordinarily the optimizer would decide the join order on fly and therefore can adjust to such changes.

Not so when the join order has been forced.

That's all for the time being.

-Pragya Agarwal 

 

Dropping Defaults (Column Constraint)

There were some recent changes that were done in regards to column defaults. In this blog entry I want to bring out, what used to happen and what happens now.

I have discussed the case of dropping a default on a column in this post, however, in a case where you are facing problems dealing with column specific constraints(especially dropping), the appraoch I suggest below should help you with that.

What used to happen in version 3.1 and earlier?


Consider the following query:

CREATE TABLE foo (col INT DEFAULT 1, col2 INT CONSTRAINT cons DEFAULT 2);

Usually when defining defaults one would choose the way it has been done for first column “col”, but even the other options as shown for second column “col2” also works.
In version 3.1 and earlier, if at all you ever decide that you don’t want the default on “col2”, you could issue a statement like this:

ALTER TABLE foo DROP CONSTRAINT cons;

Here, the default for “col2” as declared above will be stored as a constraint with name “cons”, so you could issue an ALTER statement on the table and drop it. It worked.

CREATE TABLE foo (col INT DEFAULT 1, col2 INT CONSTRAINT cons DEFAULT 2)
INSERT INTO foo (col) VALUES (0)
INSERT INTO foo (col2) VALUES (0)
INSERT INTO foo VALUES (0, 0)
INSERT INTO foo VALUES (default, default)
ALTER TABLE foo DROP CONSTRAINT cons
INSERT INTO foo (col) VALUES (0)
 SELECT * FROM foo
                 col                col2
----------------------------------------
                   0                   2
                   1                   0
                   0                   0
                   1                   2
                   0              (NULL)


What changed in version 3.5?


In version 3.5 and onwards the constraint as declared above will not be table constraint anymore, rather the default constraint on a column will be maintained as column property at column level.

So, the statement,

 ALTER TABLE foo DROP CONSTRAINT cons

would NOT work anymore. The “CONSTRAINT cons” from the original statement will be ignored. On issuing above mentioned ALTER statement, you will get an error saying its a bad constraint ID and it cannot be found.

To drop such constraint you will have to issue the following statement (this works on all versions and is the right way to drop the defaults than going  for constraint names):

ALTER TABLE foo ALTER COLUMN col2 DROP DEFAULT

To see the column properties, you can query information_schema.columns. Look for the row with your column name, there will be two columns (COLUMN_HASDEFAULT and  COLUMN_DEFAULT), they indicate whether you have the default on that column or not and the default value for that column, if any.

If you are feeling lazy to type in this, you can also do this through GUI, by editing the table schema in Visual Studio or SQL Server. In the table schema, go to the particular column and for the default property delete the value which you had previously entered. This is equivalent to issuing the above ALTER TABLE statement, in fact, behind the scenes they both use the same statement.


Ravi Tandon
SDET, SQL Server Compact

Posted by SQLCEBLOG | 1 Comments

My First Entity Application Against SQLCE

Now that SQL Server Compact 3.5 SP1 Beta is released it's time to take the first step, making your first entity application using SQLCE as a backend.

In his previous post, Ambrish gave steps on how to install SQL Server Compact SP1 Beta, along with Ado.Net Entity Framework Beta 3 and writing a small program to query the Northwind Database via LINQ.

In this post, I will briefly walk you through what Entity Framework Beta 3 has offer to, and how can we leverage it to write different types of entity queries against SQL Server Compact.

Going by EF Definition at: http://msdn2.microsoft.com/en-us/library/bb399572.aspx

The most unique selling point in my opinion becomes the part where you can run the same program/application against any backend (assuming the database schema is same), you have to just modify connection strings. Let's say even if the schema is little different, you can get that easily fixed in your metadata files, and boom, your app works.

I am including few screenshots which will guide you on how to go by making your own application and the complete program.

Please do read the previous post (by Ambrish) carefully as I will assume that you are done with the setup and we are ready to go. For those who could get things set up I am posting some screen shots to help you with that.

First step is to include an Ado.Net Entity Data Model into your project, give it a name that makes sense, I gave it SSCNorthwind.

Add Ado.NET Entity Data Model

 

Generate Model From DB

 

The screenshot below, shows the database file that we selected, its connection string that will go into your App.config file and also the name you want to give it. For simplicity I chose NorthwindEntities.

Choosing the Connection and Saving App.Config

 

The screenshot below shows you how to include the database file that you placed in your project root for generating data model.

Select Northwind DB from your samples directory

 

The wizard below allows you to choose the tables that you want to be part of entity set, as SQL CE does not support stored procedures and views as of now, we will ignore those two options.

Choose the Tables you want. SP and Views are not supported for SQLCE

 

The red box in below shows the change that we have to make in App.Config to make this working J i.e. renaming the provider:

provider=Microsoft.SqlServerCe.Client.3.5 to provider=System.Data.SqlServerCe.3.5

This is being fixed, so in RTM version you won't have to do this manual step.

You can see that the name of Entity set is NortwindEntities, so whenever we make connection to this set, the connection string provided in App.Config file will be used.

Before:

Before

After:

After

This is how the solution explorer should look now:

Solution

If you face any error while adding Northwind.sdf file into your project, just ignore it for now.

I have tried to include some comments in my program below to help you understand what's going on.

Points to notice:

To connect to a Database using an Entity Connection, you can do it in two ways.

  1. First way is to specify your own connection string, like I have done at the start of program. Using that you can directly create an Entity Connection and work in the same way you used to work with SqlCeConnection and SqlCeCommand. That way can be seen in ESQL Query region.
  2. The second approach is using Contexts or what we call here NorthwindEntities, (remember at top we renamed Entities to this for better understanding). For using context, you can do that in multiple ways
    • First, you can give a connection string and it will create the set.
    • Second, you can also first create an Entity Connection like above and then pass it on the NorthwindEntities, which is what I have done in Object Query section.
    • And the last approach for which we created App.Config in first place. You don't specify anything at all, the entity set (NorthwindEntities) picks up your connection string from App.Config and creates a connection. LINQ to Entites Query section is done using this approach.

Program (Contents of Program.cs):

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using System.Data.EntityClient;

using NorthwindModel;

using System.Data;

 

namespace MyFirstEntityApplicationForSSC

{

    class Program

    {

       

        #region Static Declarations

 

        //SQLCE Connection string for the sample database Northwind.sdf

        static string _sqlCeConnnectionString = @"Data Source = Northwind.sdf";

       

        //entityConnectionString: If your metadata(CSDL, SSDL and MSL, or in this case EDMX) files lie in the same place as application, you can use "."

        static string _entityConnectionString =

            String.Format(@"metadata=.;Provider=System.Data.SqlServerCe.3.5;provider connection string=""{0}""", _sqlCeConnnectionString);

 

        //New Values that you might want to use for changing the data

        static string _newFirstName = "AGENT";

        static string _newLastName = "SMITH";

        static string _newCountry = "MATRIX";

 

        #endregion Static Declarations

 

        #region Main

        /// <summary>

        /// MAIN: It calls the three different functions for following type of queries

        /// Description in layman language

        ///

        /// ESQL - This takes you back to old .Net 2.0 days where you used to set T-SQL statements and execute them, just that it here works on entities now instead of tables

        /// Example Demonstrates: Simple Select Query using Entity SQL.

        ///

        /// Object Query - Working on Data Objects was never so easy. This type of queries give you the power to work on data objects and most of all intellisense helps with things you can do

        /// Example Demonstrates: Query and Updation of data and then Re-Querying

        ///

        /// LINQ to Enity - LINQ means Language Integrated Query. This is a new and powerful way of writing queries that are checked for syntax at compile time itself. So, chances of failing in query logic at runtime become minimal

        /// Example Demostrates: Query the data using default settings for connection in App.Config that was configured via EF Designer.

        /// </summary>

        /// <param name="args"></param>

        static void Main(string[] args)

        {

            Console.WriteLine("SQL CE Northwind Entity App:");

            try

            {

                EntityConnection entityConnection = new EntityConnection(_entityConnectionString); //Creating a new entity connection using the entity connection string

                entityConnection.Open();

 

                Console.WriteLine("\nESQL Query:");

                ESQL_Query(entityConnection); //This does not use context, it's a simple esql command execution

 

                Console.WriteLine("\nLINQ To Entity Query:");

                LINQ_To_Entity_Query(); //This will make a new connection using the connection string in the App.Config file

 

                Console.WriteLine("\nObject Query:");

                ObjectQuery(entityConnection); //This will create the context with existing connection

 

                entityConnection.Close();

            }

            catch (Exception ex)

            {

                Console.WriteLine("\nFAIL! Oops, it was not expected, an exception has been thrown, details below:\n");

                Console.WriteLine(ex.Message);

            }

            Console.WriteLine("\nPress a Key to exit...");

            Console.ReadKey();

        }

        #endregion Main

 

 

        #region ESQL_Query

        /// <summary>

        /// ESQL_Query

        /// (Executing a simple ESQL query against Northwind Entity Set)

        /// Using the entity connection which is passed on as parameter

        /// it executes a simple entity command on the Employees entity set in the Northwind Context.

        /// The result of that query is later on printed on the screen using a data reader.

        /// </summary>

        /// <param name="entityConnection"></param>

        private static void ESQL_Query(EntityConnection entityConnection)

        {

            EntityCommand entityCommand = entityConnection.CreateCommand();

            entityCommand.CommandText = @"Select Emp.Employee_Id as Id, Emp.First_Name as Name from NorthwindEntities.Employees as Emp order by Emp.Employee_Id";

 

            EntityDataReader entityDataReader = entityCommand.ExecuteReader(CommandBehavior.SequentialAccess);

            //Note: You have to use CommandBehavior as SequentialAccess, otherwise an exception will be thrown

 

            while (entityDataReader.Read())

            {

                for (int i = 0; i < entityDataReader.FieldCount; i++)

                    Console.Write(entityDataReader[i].ToString() + "\t");

                Console.WriteLine();

            }

        }

        #endregion ESQL_Query

 

 

        #region LINQ_To_Entity_Query

        /// <summary>

        /// LINQ_To_Entity_Query

        /// (Executing a simple LINQ query against Northwind Entity Set)

        /// This function creates a new Northwind Context based on the setting provided for it in the App.Config File

        /// It executes a simple LINQ query on the Employees entity set in the Northwind Context.

        /// The result of that query is later on printed on the screen using the IQueryable structure.

        /// </summary>

        private static void LINQ_To_Entity_Query()

        {

            NorthwindEntities nwind = new NorthwindEntities(); //Uses the settings for connection string in App.Config File

 

            IQueryable<string> emps = from e in nwind.Employees where (e.Employee_ID%2)==0 select e.First_Name; //Even Number Employee Ids

 

            foreach (string e in emps)

            {

                Console.WriteLine(e);

            }

 

        }

        #endregion LINQ_To_Entity_Query

 

 

        #region ObjectQuery

        /// <summary>

        /// ObjectQuery

        /// (Executing a simple Object query against Northwind Entity Set and Updating the data)

        /// This function creates a new Northwind Context based on the entity connection thas has been passed to it as parameter

        /// It executes a simple Object query on the Employees entity set in the Northwind Context.

        /// The result of that query is later on printed on the screen and some changes are made to it.

        /// The changes are then saved using SaveChanges on Context and then re-querying them to see if things went fine.

        /// </summary>

        /// <param name="entityConnection"></param>

        private static void ObjectQuery(EntityConnection entityConnection)

        {

            NorthwindEntities nwind = new NorthwindEntities(entityConnection);

           

            var emps = nwind.Employees.Where((delegate(Employees e)

                                                {

                                                    return e.Employee_ID > 5;

                                                }

                                            ));

           

            //var emps = nwind.Employees.Where("it.Employee_ID > 5"); //This query is also equivalent to above one and will give same results

            //var emps = nwind.Employees.Where("it.Employee_ID > @empid",new ObjectParameter("@empid",5)); //This query is also equivalent to above one, but it uses parameters for the values

 

            foreach(Employees e in emps)

            {

                Console.WriteLine(e.Employee_ID + "\t" + e.First_Name + "\t" + e.Last_Name + "\t" + e.Country);

                if (e.Employee_ID == 10) //As soon as we encounter employee id as 10, we want to change some data

                {

                    Console.WriteLine("Changing Data");

                    e.First_Name = _newFirstName;

                    e.Last_Name = _newLastName;

                    e.Country = _newCountry;

                }

            }

 

            Console.WriteLine("Saving Data Changes");

            nwind.SaveChanges(); //Any changes made to the context are saved using this command

 

            emps = nwind.Employees; //This is simplest way to select all employees. If you happen to choose particular data you may modify the query accordingly, like nwind.Employees.Where("it.Employee_ID = 10")

 

            Console.WriteLine("Display Data Again:");

            foreach (Employees e in emps)

            {

                Console.WriteLine(e.Employee_ID + "\t" + e.First_Name + "\t" + e.Last_Name + "\t" + e.Country);

            }

 

        }

        #endregion ObjectQuery

 

        }

}

The output of the program would look like this:

SQL CE Northwind Entity App:

 

ESQL Query:

1       Nancy

2       Andrew

3       Janet

4       Margaret

5       Steven

6       Michael

7       Robert

8       Laura

9       Anne

10      Albert

11      Tim

12      Caroline

13      Justin

14      Xavier

15      Laurent

 

LINQ To Entity Query:

Andrew

Margaret

Michael

Laura

Albert

Caroline

Xavier

 

Object Query:

6       Michael Suyama  UK

7       Robert  King    UK

8       Laura   Callahan        USA

9       Anne    Dodsworth       UK

10      Albert  Hellstern       USA

Changing Data

11      Tim     Smith   USA

12      Caroline        Patterson       USA

13      Justin  Brid    France

14      Xavier  Martin  France

15      Laurent Pereira France

Saving Data Changes

Display Data Again:

1       Nancy   Davolio USA

2       Andrew  Fuller  USA

3       Janet   Leverling       USA

4       Margaret        Peacock USA

5       Steven  Buchanan        UK

6       Michael Suyama  UK

7       Robert  King    UK

8       Laura   Callahan        USA

9       Anne    Dodsworth       UK

10      AGENT   SMITH   MATRIX

11      Tim     Smith   USA

12      Caroline        Patterson       USA

13      Justin  Brid    France

14      Xavier  Martin  France

15      Laurent Pereira France

 

Press a Key to exit...

 

If you want your changes to persist or not to persist, modify the properties for the Northwind.sdf file in your solution properties as per your desire. By default it is "copy if newer" which means the database file in your execution folder will retain the changes. By changing it to "copy always" you can refresh your database each time.

I hope now you can now go back and start writing advanced programs using various techniques available.

Ado.Net Entity Framework is a new weapon to the list of a current developer's arsenal, opening multiple ways in which he can write his programs without worrying about the under lying intricacies.

The shift to entity plane seems very lucrative, easy and promising. And what more, all of this comes with handy tools which make life easy. Writing your code from the word Go!

The above topic was in relation to LINQ to Entities, if you are looking for LINQ to SQL refer to this blog post: http://blogs.msdn.com/sqlservercompact/archive/2007/08/21/linq-with-sql-server-compact-a-ka-dlinq-over-sql-ce.aspx

Happy Programming!

 

Ravi Tandon

SDET, SQL Server Compact

Posted by SQLCEBLOG | 8 Comments
Filed under:

Download available for SQL Server Compact 3.5 SP1 Beta for ADO.Net Entity Framework Beta 3

The SQL Server Compact 3.5 SP1 Beta for ADO.Net Entity Framework Beta 3 is available for download at Microsoft Download Center.

To install SQL Server Compact 3.5 SP1 Beta Release for ADO.NET Entity Framework Beta 3

  1. Install ADO.NET Entity Framework Beta 3 from Microsoft Download Center. You must ensure that the pre-requisites specified on the download page are installed before installing ADO.NET Entity Framework Beta 3.
  2. In Control Panel, double-click Add or Remove Programs. Uninstall SQL Server Compact 3.5 or SQL Server Compact 3.5 SP1 Beta.
  3. Install SQL Server Compact 3.5 SP1 Beta Release for the ADO.NET Entity Framework Beta 3 (SSCERuntime-ENU.msi) from the Microsoft Download Center. Note that administrator rights are required on a computer to install SQL Server Compact 3.5 SP1 Beta Release for ADO.NET Entity Framework Beta 3.

To install ADO.NET Entity Framework Tools December 07 Community Technology Preview (CTP)

  1. Install the ADO.NET Entity Framework Beta 3 and SQL Server Compact 3.5 SP1 Beta Release for the ADO.Net Entity Framework Beta 3 as specified above. Install ADO.NET Entity Framework Tools December 2007 CTP from the Microsoft Download Center. You must ensure that the pre-requisites specified on the download page are installed before installing ADO.NET Entity Framework Tools December 2007 CTP. The CTP installs on a Visual Studio 2008 installation.
  2. Copy the SQL Server Compact 3.5 SP1 Design Tools Beta (SSCEVSTools-ENU.exe) to your computer by clicking on Save or Save this program to disk and after the download is complete install SSCEVSTools-ENU.exe by double-clicking it. Do not install the SSCEVSTools-ENU.exe by clicking on the Run. While running the SSCEVSTools-ENU.exe file, Microsoft.SqlServerCe.Client.dll is extracted to %ProgramFiles%\Microsoft Visual Studio 9.0\Common7\IDE folder. The SSCEVSTools-ENU.exe will create a backup of the existing DLL at the location %ProgramFiles%\Microsoft Visual Studio 9.0\Common7\IDE.

For uninstalling SQL Server Compact 3.5 SP1 Beta refer to the read me.

The steps to create a sample application are as given below:

1.    Start Visual Studio 2008 and create a console project by clicking on the menu item File à New à Project. In the New Project window expand Visual Basic or Visual C#, select Windows and then select Console Application. For the project the .Net Framework version should be .Net Framework 3.5. Name the project as SQLCompactEDMProject. On clicking OK the project is created

2.    To generate the Entity Data Model for the Northwind.sdf, copy the Northwind.sdf from the folder %Program Files%\Microsoft SQL Server Compact Edition\v3.5\Samples to the users folder like C:\Users\<login name>\Documents\Visual Studio 2008\Projects\SQLCompactEDMProject\SQLCompactEDMProject. Select the SQLCompactEDMProject in the Solution Explorer and right-click, point to Add, and then click New Item. Select ADO.NET Entity Data Model in the Templates pane. Enter Northwind.edmx for the model name and click Add. The opening page of the Entity Data Model Wizard is displayed.

3.    Select Generate from database in the Choose Model Contents dialog box and click Next. Click the New Connection button. The Connection Properties dialog box is displayed.

4.    In the Connection Properties dialog click the button Change in the Data Source. In the Change Data Source dialog box select Microsoft SQL Server Compact 3.5 and click OK. Click the radio button My Computer in the Data Source. In the Connection Properties browse to the Northwind.sdf and click OK in the Connection Properties dialog. The Choose Your Data Connection screen in the Entity Data Model Wizard comes up.  

5.    The checkbox in front of Save entity connection settings in App.Config as: will be clicked by default. Change the name in the dialog from Entities to NorthwindEntities. Click Next to continue.

6.    The Choose Your Database Objects dialog box is displayed. By default all the tables in the database are selected and will be included in the EDM. Uncheck the box next to Tables. This deselects all the tables in the database. Expand the Tables node and select the Customers, Orders, and Products tables. Change the  Model Namespace: from Model to NorthwindModel

7.    Click Finish to complete the wizard.

a.    The wizard does the following:

b.    Adds references to the System.Data, System.Data.Entity, System.Core, System.Security, and System.Runtime.Serialization assemblies.

c.    Generates an .edmx file which encapsulates the information from the EDM mapping files.

d.    Creates an App.Config file.

8.    Double click the App.Config file in the Solution Explorer to open it. Change the provider entry from provider=Microsoft.SqlServerCe.Client.3.5 to provider=System.Data.SqlServerCe.3.5

9.    Add an Imports (Visual Basic) or using (C#) statement at the beginning of the file to reference the EDM created from the Northwind database. (The name of the model corresponds to the value of the namespace attribute in your .edmx file).

Visual Basic

Imports SQLCompactEDMProject.NorthwindModel

C#

using NorthwindModel;

10. Add the following code to the Main subroutine (Module1.vb or Program.cs)

Visual Basic

Using db As NorthwindEntities = New NorthwindEntities

Dim customers = From c In db.Customers _

Where c.City = "London" _

Order By c.Company_Name Select c

For Each c As Customers In customers

Console.WriteLine(c.Company_Name)

Next

Console.ReadLine

End Using

C#

using (NorthwindEntities db = new NorthwindEntities())

{

var customers = from c in db.Customers

where c.City == "London"

orderby c.Company_Name

select c;

foreach (Customers c in customers)

{

Console.WriteLine(c.Company_Name);

}

Console.ReadLine ();

}

11. Press CTRL+F5 to run the application. After the results display on the console screen press enter to exit the application

For more information and samples please see the ADO.NET Entity Framework Beta 3 Documentation and ADO.NET Entity Framework Samples

Regards,

 

Ambrish Mishra

Program Manager - SQL Server Compact

Posted by SQLCEBLOG | 8 Comments
Filed under: ,

SQL Server Compact Release Versions

 

This blog post is meant to give different build version numbers of our SQL Server Compact Releases. This blog post is a living one and gets updated for every release we make.

 

Before getting into the mapping, first you want to know the version you are having.  There are multiple means: 

Registry: HKLM\Software\Microsoft\Microsoft SQL Server Compact Edition\*\DesktopRuntimeVersion

File Properties: Check file properties of some SQL CE DLL and check for "File version" the first line of version tab

 

Product Name

Release Vehicle

Version

SQL Server 2005 Mobile Edition

Visual Studio 2005 RTM

3.0.5206

SQL Server 2005 Mobile Edition

SQL Server 2005 RTM

3.0.5207

SQL Server 2005 Compact Edition

Visual Studio 2005 SP1

3.0.5300

SQL Server 2005 Compact Edition

SQL Server 2005 SP2

3.0.5300

SQL Server 2005 Compact Edition

Download Center

3.0.5300

SQL Server 2005 Compact Edition

Windows Mobile 6.0

3.0.5289

SQL Server Compact 3.5 Beta1

Visual Studio 2008 Beta1

3.5.5339

SQL Server Compact 3.5 Beta2

Visual Studio 2008 Beta2

3.5.5365

SQL Server Compact 3.5 RTM

Visual Studio 2008 RTM

3.5.5386

SQL Server Compact 3.5 RTM

Download Center

3.5.5386

SQL Server Compact 3.5 SP1 Beta

SQL Server 2008 CTP5 OR

SQL Server 2008 Nov07 CTP

3.5.5608

SQL Server Compact 3.5 SP1 Beta for ADO.NET Entity Framework Provider Beta

Download Center

3.5.5626

SQL Server Compact 3.5 SP1 Beta

SQL Server 2008 CTP6 OR

SQL Server 2008 Feb08 CTP

3.5.5626

SQL Server Compact 3.5 SP1 RTM

Visual Studio 2008 SP1 RTM

3.5.5692

SQL Server Compact 3.5 SP1 RTM

SQL Server 2008 RTM

3.5.5692

 

 

 

 

 

Update: This information is now available as a KB Article 950550.  The KB article list only the final release versions and not beta/CTP/alpha versions.

 

Thanks,

Laxmi Narsimha Rao ORUGANTI

 

Connectivity Cross Version Compatibility (SQL Server Compact 3.5)

Connectivity Cross Version Compatibility

 

This blog post explains the Merge Replication connectivity cross version compatibility scenarios for the SQL Server Compact 3.5 release. For more information about the connectivity cross version compatibility scenarios in SQL Server Compact 3.1, see SQL Server Compact 3.1 Connectivity Tools.

 

This post is not applicable/refer-to Sync Services for ADO.NET.  Please refer to this post by Mr. Steve Lasker to know the differences between Sync Services for ADO.NET and Merge Replication.

 

 

Product Version Information:

The following table demonstrates the product names and code names used throughout this blog post.

 

Version

Product Official Name(s)

Product Codename

Nickname

Client 2.0

SQL Server 2000 for Windows CE

SQL Server Compact 2.0

<None>

SQL CE 2.0

Client 3.0

SQL Server 2005 Mobile Edition

SQL Server Compact 3.0

<None>

SQL CE 3.0

Client 3.1

SQL Server 2005 Compact Edition

SQL Server Compact 3.1

<None>

SQL CE 3.1

Client 3.5

SQL Server Compact 3.5

<None>

SQL CE 3.5

Server 8.0

SQL Server 2000

Shiloh

SQL Server 8.0

Server 9.0

SQL Server 2005

Yukon

SQL Server 9.0

Server 10.0

SQL Server 2008

Katmai

SQL Server 10.0

 

Summary:

 

1)    SQL Server Compact 3.5 can synchronize with only SQL Server 2005 not with SQL Server 2000.

2)    SQL Server Compact 3.5 Server Tools package enables synchronization between SQL Server Compact client versions 3.0, 3.1, and 3.5 and SQL Server 2005 through SQL Server Compact3.5 IIS components.

3)    SQL Server Compact 3.5 SP1 can synchronize with both SQL Server 2005 and SQL Server 2008. 

4)    SQL Server Compact 3.5 SP1 Server Tools package enables synchronization between SQL Server Compact client versions 3.0, 3.1, 3.5 and SQL Server 2008 through SQL Server Compact 3.5 SP1 IIS components.

 

 

Server Tools Release Plan:

Our release plan for server tools package is as follows: In Phase1, we are releasing SQL Server Compact 3.5 Server Tools package for SQL Server 2005 (Yukon) and SQL Server Compact 3.5 SP1 Beta Server Tools package for SQL Server 2008 (Katmai). In Phase2, we will release the SQL Server Compact 3.5 SP1 Server Tools package for both SQL Server 2005 (Yukon) and SQL Server 2008 (Katmai). This new package will replace the Phase 1 packages.

 

Important:

This blog post does not describe the following deployment scenarios:

1)    Upgrading clients from SQL Server Compact version 2.0 to versions 3.0 and 3.1

2)    Upgrading from SQL Server 2000 to SQL Server 2005

3)    Mixed installations including SQL Server compact client versions 2.0, 3.0,  3.1  and SQL Server or publisher versions SQL Server 2000 (8.0), and SQL Server 2005 (9.0)

For more information about these deployment scenarios, please see SQL Server Compact 3.1 Connectivity Tools documentation.

 

 General Information for Deployment:

1)    You can always install SQL Server Compact client versions 3.0 and 3.5 side-by-side.

2)    The version of SQL Server IIS components must be equal to or lesser than the version of  SQL Server Publisher.

3)    You can install the SQL Server IIS components side-by-side . When installing side-by-side, the SQL Server IIS components should be in two different physical and virtual directories.

4)    SQL Server IIS components and SQL Server Compact IIS components reside in IIS server.

5)    You can install the SQL Server IIS components from the SQL Server Installation CD. In feature selection page, choose Shared Features, and then Client Tools to install the SQL Server IIS components.

 

The new rules that apply only to SQL Server Compact v3.x:

1)    The version of SQL Server Compact v3.x client components must be equal to or lesser than the version of SQL Server Compact v3.x IIS components. In other words, SQL Server Compact 3.0 client can connect to 3.0, 3.1, 3.5 IIS components. SQL Server Compact 3.1 client can connect to 3.1, 3.5 IIS components. SQL Server Compact 3.5 client can connect to 3.5 IIS components.

2)    SQL Server Compact 3.x IIS Components are not side-by-side installable.

1     Cross Version Compatibility

1.1       Cross Version Compatibility Support Matrix

 

The following table provides the cross version compatibility support for each different type of deployment scenario. The table includes:

·         “Dx”: An identifier for each different scenario.

·         “Existing Support”: A deployment that is already supported. For more information, see SQL Server Compact 3.1 Connectivity Tools.

·         “New Support”: A new supported deployment. This new deployment is explained in this blog post later.

·         “Pending Support”:  Specifies that SQL Server Compact 3.5 RTM does not support synchronization to SQL Server 2008 but this support will be enabled in SQL Server Compact 3.5 SP1.

 

</

Deployment Scenario No

SQL Server Compact

Client Version

SQL Server Publisher Version

Supported

(Yes/No)

Notes

D1

2.0

N/A

N/A

N/A

D2

3.0/3.1

SQL Server 2000 SP3a or later

Yes

Existing Support

D3

3.0/3.1

SQL Server 2005

Yes

Existing Support

D4

3.0/3.1

SQL Server 2008

Yes

New Support

D5

3.5 RTM or later

SQL Server 2000

No

Retired/Deprecated