SQL Server Compact - Compact & Capable

Merge replication vs. Sync Services for Compact

2009-11-09T14:12:00Z

SQL Server Compact supports predominantly two different sync technologies. “Merge replication” and “Sync Services”. Users might be using one or the other, and interested in knowing the differences between the two technologies, from a use-case point of view. Still, there might be others interested in knowing, which technology better suites their need, by comparing and choosing the right one. One simple rule of thumb here: Merge replication is designed keeping enterprises in mind, where Sync Services is a framework for developer community/social networking/hobbyist programmers etc… (everything else). “Merge replication” is a solution, which is suitable for Enterprises mostly, in preference to Sync services. However, Sync services is an open-ended, extensible framework on top of which, one can do almost everything that Merge replication does. This article concentrates mostly on the merge replication angle in terms of where it makese more sense and where it doesn't.

Where does "Merge" make more sense?

In the enterprise scenario. In my opinion, enterprises like things faster, better (without any issues), and easily do-able/repeatable. If a solution (like “Merge replication”) allows them to do things like “setting up a publication, subscribers etc…”, although with significant coding, people are not going to like it. So, tools and wizards for enterprises are mandatory (Also give them a way to re-do the setups, typically outputting scripts from these tools/wizards). To avoid the setup/upgrade glitches, provide an out-of-the-box integration story, with other often used components (like Windows, IIS, SQL Server, HTTP etc…). Also, tighter integration helps performance. Usually, in enterprise contexts, setting up a “data server and clients” is a long-term commitment, so, tighter integration is fine there. All these are basic requirements for the “Enterprise” oriented “merge replication” technology.

Note: There are many ways in which a tighter integration with other components helps deployments.

1. Efficiency/performance

2. Automation possibility and tooling

3. Better end-2-end support/solution story for the customers

4. Ability to monitor and troubleshoot parts of the system

5. Essentially, you leave all the hardwork to us, and in the end, expect something that automatically works J (Means, reduced cost of development and testing for you)

Also, my take is that “Enterprises like rich (and relevant) features”. Give them something that can easily model/extend their business logic, and that could be of great value to them. Some of the features of “Merge replication”, in this category are,

1. Automatic partition management

2. Custom conflict resolution hooks

3. Business logic plugins to do custom processing, in the process of merge replication

a. Example is, if one wants to charge the client for synching

b. One wants to compute his share whenever a salesman syncs to the server

4. Retention cleanup, which can be used to (weakly) set a stringent policy about incremental syncs.

a. Like, every salesman must sync at least once in a week or every branch office, must sync at least once a day.

5. Incremental schema changes replication

6. Integration with SQL Server mirroring

7. Integration with SQL Server backup and restore.

Other valuable features are, “Automatic identity range management” etc. I hope now it is very clear as to, why “merge replication” technology is good for enterprise businesses. Enterprises do not mind, buying into a special architecture, if that provides value. Enterprises can setup a particular set (at the least) of windows machines, with IIS and SQL Server, and run merge replication. So, a tailored solution, like merge replication is best suited for them.

Where does Sync play better role compared to Merge?

Now, let’s examine another use-case. Incidentally, in this use-case, the features of merge replication, do not seem much relevant. You want to write a stock-alert application, and link it with a stock-tick source on a website. On closer examination, this is also a “data sync” scenario, so, merge replication could be used here. Trying to use merge replication here, elicits many incompatibilities/redundancies, that are not relevant to this use-case.

First of all, the nature of the data source is unknown, it can change while the app is running too. The web-site might use SQL backend now, and something else tomorrow. The transport is not known, it is over HTTP, but, can have any format (plain HTML/Json etc…). There is no need of rich data semantics here, as one is just comparing a single piece time-series data, and deciding to throw the alert or not. Also, one does not need a big database at client, all he does is read and discard (or read, alert and discard). Clearly, using merge replication here is an overkill, and unsuitable. So, you should use “Sync services”.

If you want to write a very quick sync app, and use it anywhere*, “Sync services” is the answer. Because Sync Services, is not tied to any particular, server architecture or transport mechanism, one can use it anywhere*. For quick development of sync apps, Sync Services is integrated with Visual Studio. But, this is only to develop the app quickly, not to deploy it. Visual Studio is not even needed for plain development of any sync app. It is just there to make it easy and fast.

Sync services is amazingly extensible, componentized, customizable. While merge replication answers enterprise use-case, which is a typical use-case for data synchronization, virtually all other use-cases are addressable by “Sync services”. If for some reason, people do not want to use a particular server/transport architecture, (or can’t use a single architecture), Sync services is the way to go. Besides, Sync services is a free platform to develop apps on. It is a great tool to enable non-(traditional)-enterprise related, businesses/users to realize the new models of interaction cropping up almost everywhere now. Sync services can be made to have feature-parity with “merge replication” solution through coding, although, I hope you would agree that it is not the intent of Sync services. I quoted that here, only to make you realize the potential of Sync services platform.

When an open-ended sync solution needs to be developed, Sync Services Framework is the one to use. Many times in usage, applications to sync data, need heterogeneous servers/transports etc. This is a very important capability, for example, for news generation and propagation web-site. News could be present in various web-sites, and also inside documents/databases of different formats. Also, the solution should be open-ended and extensible, so that, it can be easily tailored for a new data source. The importance of “Sync services” in enabling such scenarios, should not be underestimated. Sync Services, is coming out strong, and there is more to watch out in this field, going forward. With this, we conclude the explanations of the technologies, from the use-cases point of view.

The list of features of merge replication and Sync services are tabulated below:

Feature	Supported in merge replication	Supported in Sync Services
Enterprise-centric
Type of the technology	Solution	Framework
Target users	Enterprises with DBAs	Developers/Hobbyists/Social networking Communities
Integration with DBA tools (SQL Server)	Yes	No
More tooling support	Yes	No
Pluggable business logic hooks	Yes	No
Conflict resolution support	Built-in + support for custom***	Built-in + extensible**
Schema propagation support	Initial + any changes	Limited, available by extension**
Partitioning support	Built-in	Available by extension
Server type (State-full or State-less)	State-full	State-less
Integration with Server mirroring (SQL Server)	Yes	No
Integration with Server backup/restore (SQL Server)	Yes	No
Identity ranges management	Yes	No

Developer centric
Type of the technology	Solution	Framework
Target users	Enterprises with DBAs	Developers/Hobbyists/Social networking Communities
Developer platform support	No	Yes, Visual Studio integration provided
Pluggable transport	No (HTTP with IIS server only)	Yes (Web service model is possible, by extension)
Heterogeneous server	No (Only SQL Server)	Any server is good (by extension).
Network architectures	Fixed, 3-tier	Variable, 2-tier to N-tier
Supports web services model	No	Yes
Ability to work with other sync platforms/frameworks	No	Yes (by extension)
Exposed API surface for tracking****	No	Yes

Common functionality provided
Subscribed database deployment	Yes	Yes
Type of change tracking used	ROWGUID	PK or ROWGUID
Sync directions allowed	All	All
Programmability layer	Native & managed	Managed

Semantics provided
Auto-management of dependent tables	Yes	No
Sync granularity enforced	Yes (Publication level)	No (table level)
Schema propagation	Yes	Limited extent
Built-in conflict resolution strategies (also customizable)	Yes	Only some limited number of built-ins

* Well, almost everywhere.

** Whenever we use, “available by extension” or simply “by extension”, that means developer should write code to achieve the desired behavior.

*** (For this document,) the difference between customizable and extensible is the following: customization is a type of extension, where, the application architecture is not affected majorly. Like, register a COM dll, to do conflict resolution etc…

**** Tracking is a mechanism used to tag all changes. Changes could be Inserts/Updates/Deletes on tracked tables. This module is used to detect changes between successive syncs, so that the data can be forwarded to the other party. Exposing the tracking API (enable/disable tracking etc…), helps write applications like peer-peer sync, transaction notifications etc…

Wrap-up:

We have looked at the use-cases where merge replication technology is suitable and where it is not. There are scenarios where “Sync services” comes out as a candidate for solution, and there are scenarios for merge replication too. When considering the right technology for using, there is no silver bullet; there is no panacea that works for all needs. The requirements of your use determine the “right” technology for you. Merge replication provides a solution, but, there are many scenarios that it can’t help you with. Sync services can be made to work in “any” scenario (including the one that merge replication provides out-of-the-box J), but, are you fine with building such a (mammoth) architecture yourself, and incurring the various costs involved? In merge replication, we take the pains and give you a solution, although, for a special, tailored need, is also happens to be the most common and justified one. In sync services, you are on your own, but, it gives many benefits that you can’t do without in many situations.

Contributor: Udaya Bhanu Goteti

SQL CE Binaries - Details

2009-10-07T11:27:00Z

Many times developers get puzzled on which exact set of binaries one should consume in their application. There is also a misconception that SQL CE engine is too big in size. This may be because there was no effort from our side to clarify on “Which binary is for what purpose?”.

In an effort to get clarity in these aspects, I have published a blog post on my blog. Please do read and provide your comments.

Thanks,

Laxmi Narsimha Rao Oruganti

Troubleshooting: Access Violation exception while using SQL Server Compact database with ADO.NET provider

2009-05-06T06:26:00Z

Possible causes for Access Violation exception in general:

An access violation occurs in unmanaged or unsafe code when the code attempts to read or write to memory that has not been allocated, or to which it does not have access. This usually occurs because a pointer has a bad value.

In programs consisting entirely of verifiable managed code, all references are either valid or null, and access violations are impossible. An AccessViolationException occurs only when verifiable managed code interacts with unmanaged code or with unsafe managed code.

For more information about this, please visit this link.

When this exception can be related to SQL Server Compact:

SQL Server Compact ADO.NET provider DLL [System.Data.SqlServerCe.dll] is a managed DLL which interacts with other native DLLs to provide the functionality.

There are two known issues related to the usage of SQL Server Compact provider which may lead to this Access Violation exception:

1. Version Mismatch [3.5 RTM and 3.5 SP1]:

Access Violation occurs when the loader loads different versions of Managed and Native SQL Server Compact 3.5 DLLs. This issue arises when different versions are available in the box and this happens with Private and Central deployment of SQL Server Compact 3.5.

To be specific, if SQL Server Compact 3.5 RTM is pre-installed [centrally deployed] in the box and application built on top of SQL Server Compact 3.5 SP1 privately deploys the SQL Server Compact DLLs.

In this case, the managed DLL [System.Data.SqlServerCe.dll v3.5.5692.0] will be loaded from private directory but the native DLLs [sqlceme35.dll, sqlceqp35.dll, etc. v3.5.5386.0] might get loaded from the central deployed directory.

If the application is built against SQL Server Compact 3.5 RTM and SQL Server Compact 3.5 SP1 is centrally deployed in the box then the application will start using SQL Compact 3.5 SP1 as there is a redirection entry which redirects an application (Compiled against 3.5 RTM) to use centrally deployed SQL Compact 3.5 SP1.

2. Thread Safety:

SQL CE objects are not thread-safe and are not thread affinitive either. If an instance of SqlCeConnection or SqlCeTransaction is shared across threads without ensuring thread safety, then that may lead to Access Violation exception.

Work-around if the cause is related to SQL Server Compact:

For the above two issues, there are workarounds available which should be followed to avoid any Access Violation exception.

1. Version Mismatch:

One should follow either of the given below approaches to avoid the Access Violation exception:

Scenario: Application built against SQL Server Compact 3.5 SP1 [to be deployed privately] and SQL Server Compact 3.5 RTM centrally deployed.

Solution:

1. There should be only one version of SQL Server Compact [v3.5 RTM or v3.5 SP1] installed in the box. The version with which the application is not built against [v3.5 RTM] should be un-installed.

2. Install SQL Server Compact [v3.5 SP1] centrally to avoid any future install mismatch.

Forum thread related to this issue can be found here.

2. Thread Safety:

It is recommended that each thread should use a separate connection than sharing across. If there is really a need for sharing SQL CE objects across threads, then the application should serialize access to these objects.

Thanks,

Mohammad Imran.

SQL Server Compact Garbage Collection – Whys and Hows

2009-05-05T17:52:00Z

Some background about SQL Server Compact:

SQL Server Compact is an embedded database implemented in native and it can be accessed in application by either OLEDB or ADO.NET provider model.

ADO.NET provider for SQL Server Compact is a managed assembly which depends on native SQL Server Compact DLLs to provide the service. In general, the managed classes under ADO.NET provider model for SQL Server Compact are just wrappers around the native classes.

SQL Server Compact native objects are dependent. For instance, Cursors and Transactions are dependent on Connection object and are expected to be released in the right order.

What’s the problem then? In native application, the developer can dispose the cursors, commands, etc. before closing the connection but in managed application with .Net Garbage Collector there is no particular order in which these objects will be disposed.

So what do we do? Our SqlCeConnection object maintains weak references to all objects tied with it. Or to be specific, it maintains short weak references.

How does it help me in disposing the objects in order? When the connection object is getting disposed, the weak reference cache is iterated and all the objects there are disposed before the connection object getting disposed. Please note that when we say Dispose, we also mean Finalization.

How does .Net Garbage Collector comes into picture of SQL Server Compact

Simply employing short weak references to keep track of all related objects and dispose them in sequence does not solve all the problems.

What is the problem then? Sometimes the database files can remain locked in stress scenarios because not all the dependent objects are disposed upon dispose of SqlCeConnection. This happens because we use short weak refs to track object lifetime and we could end up in situations in which an object is in the finalization queue (hence the short reference is not longer alive) and isn’t in the freachable queue (hence isn’t guaranteed to be picked up by the GC immediately). More information about short weak references can be found here.

All this leads to a situation, where the database file is still locked even though customer application closed the connection. Hence, it is not able to get deleted the file using file explorer …etc means.

How did we solve the problem? When we are cleaning weak reference cache in dispose of SqlCeConnection object, we need to call GC.WaitForPendingFinalizers() before we return to the caller; in this way, we are guranteed that the GC will pick up all finalizable objects and hence all the native interfaces will be properly released even if we no longer have short references to them.

More information about this API can be found here.

FAQs

1. Why are we employing short weak references and not long weak references?

Ans.: SQL Server Compact ships for both Desktops as well as Devices. Long weak references are not supported in .Net CF. More information here.

2. Does call to GC.WaitForPendingFinalizers() has negative performance impact?

Ans.: No. Most of the time, the freachable queue is empty (the same is not true for finalization queue though). See here for more information about Garbage Collector.

3. Can this call to GC.WaitForPendingFinalizers() lead to deadlocks?

Ans.: Yes. If Dispose method of SQL CE objects are called in the finalization context than the explicit dispose context. The reason being when an object is getting finalized, it is not supposed to touch any other managed objects as the currently getting disposed object does not know the life-status of the object it is trying to refer.

4. Why did we choose this design?

Ans.: SQL CE was primarily designed for Devices where the memory, processing comes at very high cost. So, it has been designed to free up the resources as early as we can and hence it is calling GC.WaitForPendingFinalizers.

Thanks,

Mohammad Imran.

After moving the database from one platform to other, the first SqlCeConnection.Open() takes more time

2009-04-01T15:16:00Z

If you move a SQLCE database from one platform to other, it's first SqlCeConnection.Open() takes more time and also increases the database file size.

The Reason: For an index on string type columns, SQLCE uses LCMapString API to get the normalized sort key. LCMapString API behavior will be different for different NLS sort versions. NLS sort version is different for Desktop Windows OS and WM/WCE. During the first database connection open, SQLCE rebuilds all indexes if there is a NLS sort version mismatch with the last accessed OS platform. Index recreation requires space on database file and hence the database file size will be increased. You can use Compact()/Shrink() API to get the database file with approximately old file size. Because of this index recreation, first database connection open takes more time as it has to rebuild all indexes.

Note: If your database doesn’t have any indexes you will not see above mentioned symptoms on first connection open.

Don’t inter-op SQLCE 3.0 databases between devices and desktop:

SQLCE started supporting desktops only from SQLCE 3.1 (3.0.5300.0) release. SQLCE 3.0 doesn’t have index rebuilding logic and the index re-creation was added during SQLCE 3.1 (3.0.5300.0) release as it is required to support platform inter-op for SQLCE databases. On desktop you can open SQLCE 3.0 databases, which were created on devices. Again If you open these databases on SQLCE 3.0 devices, your device app might receive an error/exception.

For example: Opening a SQLCE 3.0 database with SSMS will use SQLCE 3.1 bits on desktop and hence it would drop and recreate index data. When this file is copied back to device, index data may not be valid and receive an error/exception.

Namaskaram!

Manikyam Bavandla

Overview of SqlCeReplication methods - LoadProperties and SaveProperties

2009-02-06T18:22:00Z

Whenever sync is started, developers provide all the necessary properties to an object of SqlCeReplication class. These properties are provided through the constructors or by directly setting the public properties of the object. In this approach developers need to provide this data for every sync, which forces them to store this data somewhere in their code. Even if the properties seldom change, developers provide this data all the time. This data includes passwords such as DistributorPassword, InternetPassword, InternetProxyPassword and PublisherPassword which developers are better off, not saving or handling in their code.

This blog article throws light on SaveProperties/LoadProperties methods which provide a good solution to issues described above. SaveProperties saves all the properties a developer can provide to SqlCeReplication object (except for two which are described below) in a system table called __sysMergeSubscriptionProperties within the replicated database. The benefits of this approach are:

1) The replicated database can be encrypted securing this data inside the sdf file.

2) Passwords are encrypted with the unique device hardware key enabling an extra layer of security.

3) Data that seldom changes need not be explicitly provided by developer for every sync

Code example for calling SaveProperties:

// Example code for SaveProperties starts

repl = new SqlCeReplication();

repl.InternetUrl = "http://www.adventure-works.com/sqlmobile/sqlcesa35.dll";

repl.InternetLogin = "MyInternetLogin";

repl.InternetPassword = "<enterStrongPassword>";

repl.Publisher = "MyPublisher";

repl.PublisherDatabase = "MyPublisherDatabase";

repl.PublisherLogin = "MyPublisherLogin";

repl.PublisherPassword = "<enterStrongPassword>";

repl.Publication = "MyPublication";

repl.Subscriber = "MySubscriber";

repl.SubscriberConnectionString = "Data Source=MyDatabase.sdf";

repl.SaveProperties();

// Example code for SaveProperties ends

Note: SaveProperties will succeed only if the Subscription is already created on the machine.

This simplifies subsequent syncs in that developers need to provide only SubscriberConnectionString (to identify the local database) to SqlCeReplication object and call LoadProperties. If __sysMergeSubscriptionProperties table has only one record this data is automatically loaded into SqlCeReplication object. If there are more records, then in additional to SubscriberConnectionString, developers need to provide Publication, Publisher and PublisherDatabase to uniquely identify a saved profile for a subscription.

Code example for calling LoadProperties:

// Example code for LoadProperties starts

repl = new SqlCeReplication();

repl.SubscriberConnectionString = "Data Source='Test.sdf'; Pwd='<enterStrongPassword>'";

repl.LoadProperties(); // assuming only one subscription’s properties are saved

// Example code for LoadProperties ends

LoadProperties/SaveProperties thus provide an efficient way to store and retrieve data relieving developers from maintaining sensitive and unchanging data.

Properties not saved by SaveProperties:

SendTimeout and ReceiveTimeout properties are not saved via SaveProperties. These values must be explicitly provided to SqlCeReplication object for every sync.

Note:

There are two things to note while using this functionality.

1) Before calling SaveProperties, please make sure all necessary properties are set on the SqlCeReplication object. If we call SaveProperties with required properties missing, this data will be set to default even if there is a saved profile already present for this subscription in the Properties table.

2) Passwords stored are encrypted with the unique device hardware key. Therefore if the database file is moved to a different machine LoadProperties will fail to retrieve the correct password. Developers need to provide the password and other required properties, and call SaveProperties again to refresh the saved profile on the new machine.

For more information please refer MSDN documentation:

a) SqlCeReplication.SaveProperties - http://msdn.microsoft.com/en-us/library/system.data.sqlserverce.sqlcereplication.saveproperties.aspx

b) b) SqlCeReplication.LoadProperties - http://msdn.microsoft.com/en-us/library/system.data.sqlserverce.sqlcereplication.loadproperties.aspx

Thanks,
Balasubramaniam Bodedhala

How to: Authoring a 64-bit ClickOnce Bootstrapper package in Visual Studio 2008 SP1

2008-10-06T16:20:00Z

Before going into further details first “Disclaimer: This is just to help and does not guaranty that this is approved my employer or me J”

Location of SQL Compact Bootstrapper

<BootstrapperDir>\Packages\SQL Server Compact Edition\<LocaleCultureDir>\

Where,

· <BootstrapperDir> can be found by reading [HKLM\Software\Microsoft\GenericBootstrapper\3.5]path value otherwise its default value is %ProgramFiles%\Microsoft SDKs\Windows\v6.0A\Bootstrapper\ .

· <LocaleCultureDir> is culture specific dir, ex: EN, DE, zh-CHS, zh-CHT, …

Follow the below steps, If you have an ‘Any CPU’ ClickOnce application using SQL Compact 3.5 SP1, for which you want to install SQL Compact x64 MSI on 64-bit machine.

Note: It is required to install both x86 and x64 MSIs on 64-bit OS, since x86 MSI is pre-requirement for x64 MSI.

Steps:

1) Download the SQL Compact 3.5 SP1 x64 MSI and place it under <BootstrapperDir>\Packages\SQL Server Compact Edition\<LocaleCultureDir>\

2) ClickOnce setup.exe is always runs as WoW process on x64 OS, that’s why reading a registry key is not useful for x64 MSI. We can achieve the install check using <MsiProductCheck> element.

Edit the <BootstrapperDir>\Packages\SQL Server Compact Edition\<LocaleCultureDir>\Package.xml file with following details

· As part of <InstallChecks> element add <MsiProductCheck> as below

---

---

</InstallChecks>

[ProductCode] is the Product Code/GUID of the x64 MSI. Depending on locale, get it from below table for 3.5 SP1 x64 release.

You can use Orca.exe to get the ProductCode property of a MSI.

Locale	Three letter locale Id	Product Code
Chinese	CHS	{8DD60183-76ED-4416-8C9C-E5A39E1826EF}
Chinese (Taiwan)	CHT	{A423B3FB-C9E6-4953-9A83-2A5F45CAF466}
German	DEU	{77CB2F9F-67C5-4ADA-9321-B30C9C64727E}
English	ENU	{F83779DF-E1F5-43A2-A7BE-732F856FADB7}
Spanish	ESN	{5B32AC72-6251-47F4-BD1B-AD479E3EEBA9}
French	FRA	{A64CF374-A3DA-4B1E-A42A-6394C48F431A}
Italian	ITA	{4634B103-984E-4F31-BD80-DCD83AEEEF85}
Japanese	JPN	{1A22CAF6-E6FD-4D65-AEBA-F28D23B68EBF}
Korean	KOR	{38514244-6C25-42EC-B144-276F6DDAC9CE}
Portuguese (Brazil)	PTB	{C8445DBB-783F-4804-B373-D5CDC0614E60}
Russian	RUS	{CAA59A81-E35A-4582-80FF-F19520FFF60F}

3) Add <PackageFile> element for x64 MSI. Replace [ThreeLetterLocaleID] with three letter locale id mentioned in above table.

---

</PackageFiles>

4) Add <Command> elements for x64 MSI.

</Commands>

</InstallConditions>

</ExitCodes>

</Command>

</InstallConditions>

</ExitCodes>

</Command>

</Commands>

5) For making AnyCPU SQL Compact 3.5 SP1 Clickonce app to download 64-bit and 32-bit MSIs from microsoft download site (In this case, you can skip step 1), you need to do following additial things.

a) Add HomeSite attribute to the 64-bit <PackageFile> element.

---

<PackageFile Name="SSCERuntime-[ThreeLetterLocaleID]-x64.msi"

HomeSite="HomeSiteName_64"

PublicKey="3082010a0282010100a2db0a8dcfc2c1499bcdaa3a34ad23596bdb6cbe2122b794c8eaaebfc6d526c232118bbcda5d2cfb36561e152bae8f0ddd14a36e284c7f163f41ac8d40b146880dd98194ad9706d05744765ceaf1fc0ee27f74a333cb74e5efe361a17e03b745ffd53e12d5b0ca5e0dd07bf2b7130dfc606a2885758cb7adbc85e817b490bef516b6625ded11df3aee215b8baf8073c345e3958977609be7ad77c1378d33142f13db62c9ae1aa94f9867add420393071e08d6746e2c61cf40d5074412fe805246a216b49b092c4b239c742a56d5c184aab8fd78e833e780a47d8a4b28423c3e2f27b66b14a74bd26414b9c6114604e30c882f3d00b707cee554d77d2085576810203010001" />

</PackageFiles>

b) Add <String> for HomeSiteName_64 in <Strings>

---

<String Name="HomeSiteName_64">[x64MSIDownloadURL]</String>

---

</Strings>

In place of [x64MSIDownloadURL] put download URL for locale specific x64 Runtime MSI.

Below is the list of locale specific download URLs for x64 3.5 SP1 Runtime MSIs.

English: http://download.microsoft.com/download/8/4/2/8423C019-CCB4-4D7D-B7F0-BCF83F1B9218/SSCERuntime-ENU-x64.msi

German: http://download.microsoft.com/download/1/5/1/1517BA39-EAC3-4281-900B-AFB77369169A/SSCERuntime-DEU-x64.msi

Spanish: http://download.microsoft.com/download/C/B/E/CBE7CA6E-0D13-4186-A3CD-03C74CE9A0FA/SSCERuntime-ESN-x64.msi

French: http://download.microsoft.com/download/3/A/3/3A3B694E-9742-4AFF-8357-DCC9BAAE8A40/SSCERuntime-FRA-x64.msi

Italian: http://download.microsoft.com/download/9/E/F/9EF6D714-80EA-4373-962C-60436ED8A1D8/SSCERuntime-ITA-x64.msi

Japanese: http://download.microsoft.com/download/D/7/1/D717C79C-D847-42BF-BB9C-C1C390D203C4/SSCERuntime-JPN-x64.msi

Korean: http://download.microsoft.com/download/B/5/7/B5773FB4-1593-4746-824D-15235D820EFC/SSCERuntime-KOR-x64.msi

Portuguese (Brazil): http://download.microsoft.com/download/D/3/8/D388E5C0-8671-4F6A-8A05-86B9470B4E97/SSCERuntime-PTB-x64.msi

Russian: http://download.microsoft.com/download/2/3/E/23EB9CB1-F037-463F-BE83-AD3848FD1480/SSCERuntime-RUS-x64.msi

Chinese: http://download.microsoft.com/download/9/A/0/9A04CFE5-29CB-4B5C-A5F9-C66CB5FD1A9B/SSCERuntime-CHS-x64.msi

Chinese (Taiwan): http://download.microsoft.com/download/D/E/B/DEB711BD-FE6A-45E9-AA15-D9901E8458C3/SSCERuntime-CHT-x64.msi

ClickOnce Reference: http://msdn.microsoft.com/en-us/library/ms229223(VS.80).aspx

Thanks

Manikyam Bavandla

Sync Services optimizations

2008-10-06T15:40:00Z

There are some optimizations that can be done for an "Sync Services" sync scenario. These are, (might not be an exhaustive list)

Small sync chunks (using smaller SyncGroups which are only absolutely necessary)
Index creation on tracking and filtering columns
Transfer encryption of data payload on web
Batching the changes
Filtering the data at various stages to reduce network load

    Sync (synonym for "Sync Services sync", in this post) interfaces with the providers using DataSet objects. That is, sync uses DataSet to store and retrieve changes happening at client and server. DataSets are known to be not very memory efficient, and the amount of memory consumed by these, is a bit of a concern when one is using these in device clients in Sync Services.

    Sync happens in chunks, using the SyncGroups that one supplies. That is, all data in a SyncGroup is logically one sync, and corresponds to one DataSet. Since sync DataSets are one per SyncGroup, and these are needed only till the corresponding SyncGroup commits, we can optimize the memory taken by DataSets, using smaller SyncGroups. Divide sync tables into various SyncGroups, so that, each group has it’s own (smaller) DataSet, and it is disposed (by garbage collector) once the SyncGroup changes are committed. However, to ensure logical consistency, you should always keep the tables which are related (by a foreign key constraint, for example), in the same SyncGroup.

Example:
Tables Order and OrderDetails are related.
Tables Customers and ShippingDetails are related. But, these are unrelated to the first set.

SyncGroup customerSyncGroup = new SyncGroup("Customers");

SyncTable customerSyncTable = new SyncTable("Customer");
customerSyncTable.CreationOption = TableCreationOption.DropExistingOrCreateNewTable;
customerSyncTable.SyncDirection = SyncDirection.DownloadOnly;
customerSyncTable.SyncGroup = customerSyncGroup;

SyncTable orderShipSyncTable = new SyncTable("ShippingDetails");
orderShipSyncTable.CreationOption = TableCreationOption.DropExistingOrCreateNewTable;
orderShipSyncTable.SyncDirection = SyncDirection.DownloadOnly;
orderShipSyncTable.SyncGroup = customerSyncGroup;

SyncGroup orderSyncGroup = new SyncGroup("Orders");

SyncTable orderSyncTable = new SyncTable("Order");
orderSyncTable.CreationOption = TableCreationOption.DropExistingOrCreateNewTable;
orderSyncTable.SyncDirection = SyncDirection.DownloadOnly;
orderSyncTable.SyncGroup = orderSyncGroup;

SyncTable orderDetailSyncTable = new SyncTable("OrderDetails");
orderDetailSyncTable.CreationOption = TableCreationOption.DropExistingOrCreateNewTable;
orderDetailSyncTable.SyncDirection = SyncDirection.DownloadOnly;
orderDetailSyncTable.SyncGroup = orderSyncGroup;

    Rather than having a single SyncGroup with all 4 tables, one can create two sync groups. One for the first set, and the second for the other set.

    Also, to increase the performance of enumerating process at server, one can create indexes on the tracking and filtering columns. As an example, let us say one has two tracking timestamp columns __CreateTime and __UpdateTime. Since, our server side changes enumerating queries always have filtering based on these columns, one should create indexes on these, so that, enumerating server changes becomes faster, and, sync performance increases.

    In case of device syncs, transferring the data on the network/internet is also going to cost more time/resources. Since, transfer of data from a web service/server machine to a device is time/bandwidth consuming, one should look at compressing data sent/received. This is mostly the case, when data being synched is large (Something like, initial synching SQL Compact with SQL Server). One can use System.IO.Compression or your own implementation of compression/decompression algorithm. (More details at http://msdn.microsoft.com/en-us/library/system.io.compression.aspx). Another blog post that refers to this issue and proposes solutions is at, (http://blogs.msdn.com/mahjayar/archive/2008/10/01/dbsyncprovider-improving-memory-performance-in-wcf-based-synchronization.aspx)

    Sync Services Server side supports batching. Please refer to http://msdn.microsoft.com/en-us/library/bb902828.aspx for more details.

    Sometimes, we may have server tables holding and synching data, that is alien to clients. For example, server table has a large GPS map of some place (in custom format) per row, that is associated with the data in other columns in the row. The map is stored as an image at client side. But, since, the data is of proprietary format, client can’t interpret the data, and it is not used anyway at client side. And also, since, it is large data, lot of network bandwidth is consumed and response time is compromised for transferring this across. It is better, if one can filter these columns out at server side, rather than sending to client.

    Another scenario is as follows: The client devices of a Sync application are used by Salesmen of a company, and the server side is the company office, where data pertaining to each salesman is stored in a SQL Server database. When the client devices sync, they are mostly concerned about the data relevant to them alone, and not to other sales persons. Here also, one needs to filter rows from the table.

    To facilitate filtering of rows/columns at client/server sides, one can use a number of strategies. For filtering data sent from client (or at the client side), one can use the callback architecture of "Sync Services". For server side, since you have more control there, one can choose to strip down the DataSet, when one is sending it over the wire/use callbacks to strip off DataSet data/client can supply queries to server side adapters, which filter data.

Explaining each of these options:

Callback architecture of sync: Sync provides various callbacks during the process of Synching. One can hook in the callbacks for accessing/modifying the dataset generated at every stage of sync. For example, one can add/drop some columns to the dataset, after changee enumeration is done at client side. After changing the dataset, effectively, the changes being applied to client database are changed. The same provision is present at server side also.

This callback architecture can be used to modify the DataSet applied/sent to server, on the fly. This is one type of filtering, possible at both client and server.

Server side gives more options when sending data over the wire. First of all, server chnages are enumerated within the "Sync Services" application itself. That is because, server change enumeration is done by queries supplied by the Sync Services client. So, which columns to select from server, can be controlled from the client side, when specifying queries like, SelectIncrementalInserts etc... This is another way of programmatically filtering server data.

One more option to filter data in a N-tier architecture is that, filtering the wire transferred content. In an N-tier architecture, client and server are in two different, but connected machines. Typically, client machine has a proxy server provider, which delegates calls to enumeraion/application of changes etc... to a web service. Web service talks to a backend data store, and services the data and operation requests. In this setting, interesting possibilities like, use of compression over the wire, and filtering over the wire etc... come up. When server side data store gives data, it can do some filtering. Also, the client side proxy can be used to filter irrelevant data. This is another type of filtering.

Example for callback architecture:
For client side DataSet filtering (client side), use SelectingChanges before the enumeration and ChangesSelected after the enumeration
For server’s data set filtering (client side), ApplyingChanges before the application of changes and ChangesApplied after the application
For filtering server side changes (at server side):SelectingChanges before the enumeration and ChangesSelected after the enumeration
For client’s data filtering (at server side):ApplyingChanges before the application of changes and ChangesApplied after the application

In every one of these methods, eventArgs.GroupMetadata and eventArgs.Context.DataSet are the ones need to be changed.

Thanks

Udaya Bhanu,

SDE II, SQL Compact

Released - SQL Server Compact v3.5 SP1, SQL Server 2008 RTM, Visual Studio 2008 SP1, .NET Framework v3.5 SP1

2008-08-13T17:37:00Z

Yes, all of them are released on August 11, 2008.

SQL Server Compact v3.5 SP1:

What is new in this Service Pack? - Click here and here

Where to download?

SQL Server Compact v3.5 SP1 for Desktops (32-bit and 64-bit)

SQL Server Compact v3.5 SP1 for Devices (all platforms & processors)

Server Tools (32-bit and 64-bit) (See Also: this)

Documentation, Books Online

Visual Studio 2008 SP1

What is new in this Service Pack? - Click here

Where to download? - Click here

.NET Framework v3.5 SP1 - Download it here

SQL Server 2008 RTM - Click here

The good thing is with VS 2008 SP1, you will get "SQL Server Compact SP1 for Desktop (32-bit)", and of course updated "SQL Server Compact SP1 Design Tools".

We received a huge feedback from you on VS 2005 SP1 that we should limit the Service Pack Size. With that feedback, we felt it is a reasonable to ask customers to download "SQL Server Compact v3. SP1 for Devices" explicitly than including it with Visual Studio 2008 SP1 there by bloating VS 2008 SP1 size. We also made a decision to not include "SQL Server Compact v3.5 SP1 for Desktops (64-bit)" so as to make sure we do not trouble all VS 2008 SP1 customers.

What about Platform Builder Updates?

Platform Builder 5.0 and 6.0 Updates for SQL Server Compact v3.5 SP1 are also released to Platform Builder team and they should be available in the next Platform Builder 5.0 and 6.0 Update. I will update this blog post with the download URL soon after they are available.

Windows CE 5.0 Platform Builder Monthly Update (July 2008) - Download it here

Windows Embedded CE 6.0 Platform Builder Monthly Update (July 2008) - Download it here

Thanks,

Laxmi Narsimha Rao Oruganti

Insight into SQL Server Compact database corruptions

2008-06-25T18:20:00Z

Database corruptions are often the most painful bugs to repro and debug. A good percentage of the SQL Server Compact bugs reported are in this category. However it so turns out that in majority of the cases Compact is not the culprit. Surprised!! Let us see why.

How SQL Server Compact detects a file corruption?

Compact database comprises of a series of pages, each 4K in size. When a page is written to disk the following sequence of steps is executed.

1. Compute the checksum on the page data and store it as part of the page.

2. Encrypt the page.

3. Write to disk at a given file location or offset.

4. Update system pages/run time cache with the page offset.

When a page is read from disk the following sequence of steps is executed.

1. Read from the disk.

2. Decrypt the page.

3. Re-compute the checksum and compare it with the expected value. (If it fails page is not fully baked. Meaning file write failed)

4. Check if this is the same page that we are expecting as per the system pages/ run time cache. (If this fails we lost a complete page write)

If (3) or (4) of the above read steps fail, the db is marked as corrupt and any further db activity is suspended till it is repaired successfully.

Notes:

· Verify reports step (3) errors as “Bad Checksum - idPage: %d, tyPage: %d, iPage: %d”.

· Verify reports step (4) errors as “Page with valid ordinal not found - idPage: %d, iPage: %d”

What could have triggered the corruption?

1. A simple device on/off or suspend/resume can result in half-baked file. (No reason to panic as we handle this as described in the next section.)

2. Compact relies on underlying FAT/TFAT file system to manage the database file. Any bugs in these components can lead to Compact file corruption. Compact stresses file system in many different ways as it tries to keep the file size as small as possible. As it employs shadow paging the writes and reads are quite random. (Note: 6 months back a bug has been identified in FAT/TFAT system that results in a Compact file losing some of the clusters during shrink operation.)

3. If SD card is being used for db file storage, it’s being equally stressed as the above.

4. More importantly SD cards bring in additional complexities during Suspend/Resume cycles. Here is a good blog link: http://blogs.msdn.com/windowsmobile/archive/2007/01/12/everything-you-want-to-know-about-sd.aspx

Am I suggesting SQL Server Compact bugs can’t result in any kind of corruption?

Of course Compact code bugs can result in corruption. A lot of them have been found internally before v3.0 release and also reported by customers thereafter. They are more logical in nature like index constraint is compromised or a table page says there are 1000 rows when there are only 55. We did a very good job in fixing these and they are hard to find now. However Compact bugs don’t result in physical corruptions like “checksum errors” or “page missing” errors.

What does Compact do to prevent corruptions?

We believe most of the corruptions should be due to device suspend/resume or on/off cycles. For ex: Compact is trying to write to a file and the user simply boots his device. Now file is in half-baked state. Or let’s say process gets killed before it can complete the flush cycle. How do we avoid this?

Well, we use shadow paging to overcome this situation. We can loosely categorize the db pages into user data pages, system pages and header page. As the name suggests user data pages contain all the user data. Header pages and system pages help us locate a specific user page on the disk. When it’s time to flush the changes to disk, we write shadows of the user data pages first. Then the shadows of the updated system pages are written. Finally the header page is written. The old pages are retained till the header page is updated. This ensures that we have a transactional write to the disk. Either the change is completely present or not.

Conclusion: Why do we still see corruptions?

Clearly the above is not enough. Some of the interesting cases are

· What if any of the underlying systems ‘SD card software’ or FAT/TFAT has a bug?

· Suspend/resume cycle can result in file buffer loss if SD card is used. So the changes supposed to be written to disk could be lost.

The intent is not to blame other components. However in the last 3 years, at least 90% of the corruption bugs reported are due to bad checksum. It means disk write is bad for some reason (which is not owned by Compact). The fault is somewhere in the bottom stack and it has to be fixed. Our options are very limited without a repro but we are trying hard to improve our debugging infrastructure. We are hoping that with this knowledge customers will be able to understand the corruptions better and come up with good practices that would vastly reduce these corruptions.

Thanks

Raja

Katmai new datatypes support

2008-06-25T15:31:00Z

In Katmai, new datatypes are introduced. Some of these (including some older types in Yukon) are supported for merge replication synching with SQL Compact in SQL Compact 3.5 SP1. These types are mapped to downlevel compatible types for SSC. These types and their mappings are, as follows:

SQL type Compact type Format

Date nvarchar(10) ‘yyyy-mm-dd’

Time nvarchar(16) ‘hh:mi:ss.nnnnnnn”

Datetime2 nvarchar(27) ‘yyyy-mm-dd hh:mi:ss.nnnnnnn’

DatetimeOffset nvarchar(34) ‘yyyy-mm-dd hh:mi:ss.nnnnnnn [+|-]hh:mi’

Geometry image

Geography image

The format of strings used when inserting/updating the data on SQL Compact, should be same as that given above, or as the type demands, in case of Spatial types.

Spatial types are interesting, because, one can map them to image columns on the client without data loss. The data can even be manipulated on the client side, if the required spatial type libraries are present. Other new types in Katmai, like, HierarchyID etc... are not supported for merge with SQL Compact.

Thanks

Udaya

Query performance

2008-04-30T08:15:00Z

Understanding the reason for slowness of a SQL query and then tuning to boost it is a slightly complicated process, but it can be extremely rewarding in some cases. In this post, I am going to list down some ways in which you can tune the query performance for SQL Server Compact, along with pointers to relevant online resources. If you are a seasoned database developer, you probably know most of the tricks already.

Before I jump into it, an overview of the basics first.

Basics:

Displaying actual and estimated Execution plans for SQL Server Compact: You should be familiar with Execution Plans (also called as query plans and showplans).

For non-parameterized queries: There are several easy ways to do this

When database is on a docked device or desktop: Generate a query plan using SQL Server Management Studio (SSMS) that gives a nice graphical view of the query plans. (You can connect to the database on device as long as it is docked. No need to copy the file to desktop.)
When database is on a device: Generate a query plan using Query Analyzer. This generates a .sqlplan file that can be viewed graphically in SQL Server Management Studio as before (preferable) or opened as a plain XML file.

For Parameterized (as well as non-parameterized) queries:

Use the TSQL Statements SET SHOWPLAN_XML (for estimated execution plan) and SET STATISTICS_XML (for actual execution plan), followed by SELECT @@SHOWPLAN to generate the query plans. Store the results as an XML file. (Tip: To view the query plans graphically, just rename the file with a .sqlplan extension and it is ready to be opened in SSMS like an ordinary non-parameterized query plan.)

Indexes and statistics on indexes:

One of the important tasks of the query optimizer is to select the right index to execute the query. An optimizer can make better decisions if it has histogram data (i.e. statistics) about the distribution of values for an index.

CREATING/UPDATING/DELETING statistics on an index: SQL Server Compact 3.5 creates statistics on all indexes by default and updates them as required. (You can surely tweak these settings, but it is advisable not to. See CREATE STATISTICS /UPDATE STATISTICS/DROP STATISTICS).
To view the statistics distribution, the built in stored procedure sp_show_statistics is pretty handy.
Statistics were not created by default on primary key indexes till version 3.1, so you might want to check this and add them yourself :- ).

... A lot of other things too J, but this will do for the time being.

The Meat:

If you are trying to improve the performance of your application that uses SQL Server Compact, make sure you have checked the following resources on MSDN already (In this post I will not go into any of them):

Database Design and Performance: This page talks in some detail about the following:

Database denormalization
Variable and fixed sized columns
Effect of row and key lengths

Query Performance Tuning: This page explains some of the following:

Improving indexes by creating selective indexes, choosing the columns in a multi-column index, issues in indexing a small table etc.
When to create indexes
Using parameterized queries where possible.
Rewrite queries where possible
Using base table cursor (CommandType.TableDirect) mode for simple queries.

(The article on Query Process Overview and Performance Tuning approaches for SQL Server CE 2.0 covers the same points as above, but I found it better than the MSDN links J. It is relevant for SQL Server Compact 3.5 too.)

Now here are some additional ways you can improve performance:

1. Recompile Stale query plans

<StoryTime>Sometime back I got a complaint about slow performance of our database even though the user was using parameterized queries and had created the correct indexes everywhere. The sequence in which he was performing operations was something like this:

- Create the full database schema (tables, indexes etc).

- Prepare SqlCeCommand objects for *all* queries that will ever be used by the application.

- Populate the database with actual data. Until now all tables were empty.

- Execute the queries using previously prepared command objects

What's wrong with this?

A prepared command object (SqlCeCommand) holds a query plan. The query optimizer generated this query plan and had optimized it for the state of database when the plan was generated. In this user's case, the database had just empty tables at that point in time. The optimizer therefore inferred that using a table scan is the best way of executing the query (as opposed to using any index). As a result, all his query plans used just table scans instead of the right indexes even when the table size increased!

</StoryTime>

Moral of the story: Caching query plans is good, but only as long as the state of database when they are compiled is representative of the average state of database during execution. If your data is changing rapidly, it is better to recompile the queries every once in a while.

So, beware of stale query plans!

Note that query plans can be cached for both parameterized and non-parameterized queries and you can run into this problem in either case. (Another insider hack: SqlCeCommand.Prepare doesn't really prepare the query plan. It just marks the command object for plan (re)compilation and the plan is compiled when SqlCeCommand.ExecuteXXX is called next time.)

2. Impression of using parameterized queries without really using them

Question: What's wrong (rather sub-optimal) with the following code:

SqlCeCommand cmd = con.CreateCommand();

cmd.CommandText = "SELECT * FROM table1 WHERE C1_DECIMAL = @p1";

cmd.Parameters.Add("@p1", SqlDbType.Decimal);

cmd.Parameters["@p1"].Value = 3.5;

SqlCeDataReader rdr = cmd.ExecuteReader();

while (rdr.Read()) { /* read the data */}

rdr.Close();

cmd.Parameters["@p1"].Value = 335.01;

rdr = cmd.ExecuteReader();

while (rdr.Read()) { /* read the data */ }

rdr.Close();

Answer: The query plan gets compiled twice, once for every ExecuteReader call!

Reason: First time the plan is compiled, the parameter's precision is 2 and scale is 1 (as inferred from its value ‘3.5'). The query plan therefore uses this precision and scale values. The only other values it can accept are those that fit within this range.

When the query is executed again, the parameter's precision is 5 and scale is 2 (as inferred from its value ‘335.01'). Since it doesn't fit in the parameter as inferred during first plan compilation, the plan is recompiled silently!

Solution: If you can anticipate the range (size, precision or scale) of the parameter values, then specify it explicitly. The plan will be generated based on the specified range then. This holds true for all character, binary and numeric types. (The flip side is that for any parameter value that does not fit into the specified range, an error will be thrown.) So the above code can be modified as follows:

SqlCeCommand cmd = con.CreateCommand();

cmd.CommandText = "SELECT * FROM [TABLE1] WHERE [C1_DECIMAL] = @p1";

cmd.Parameters.Add("@p1", SqlDbType.Decimal);

cmd.Parameters["@p1"].Precision = 10; // Playing safe by taking a larger range!

cmd.Parameters["@p1"].Scale = 5;

cmd.Parameters["@p1"].Value = 3.5;

SqlCeDataReader rdr = cmd.ExecuteReader();

while (rdr.Read()) { /* read the data */}

rdr.Close();

cmd.Parameters["@p1"].Value = 335.00;

rdr = cmd.ExecuteReader();

while (rdr.Read()){ /* read the data */ }

rdr.Close();

3. Query optimizer did not choose the best index

The query optimizer usually does a good job of choosing the index. However, it does it on a best effort basis and there can be cases where it doesn't pick the right index. In such cases, it makes sense to use index hints and outsmart the optimizer:

E.g. ...FROM [TABLE1] AS T1 JOIN [TABLE2] AS T2... can be rewritten with index hints as

...FROM [TABLE1] AS T1 WITH (INDEX(IX_On_Table1))

JOIN [TABLE2] AS T2 WITH (INDEX(IX_On_Table2))...

How do you know if your index is indeed better than the index chosen by optimizer?

Well, the easiest and brute force method is to run the query with and without an index hint and see which is better. I often use the Query Analyzer on device and SSMS on desktop to get the execution times. Another way is to generate query plans for both queries and study them for better predicate matches, index ordering etc.

4. Query optimizer did not choose the best join order

As in the case of choosing a wrong index, an optimizer can sometimes choose a wrong join order too. Again a corner case, but not impossible to run into. In such cases, you can specify the join order explicitly:

E.g. ...FROM [TABLE1] AS T1 INNER JOIN [TABLE2] AS T2 ON ... can be rewritten with a forced join order as

...FROM [TABLE1] AS T1

INNER JOIN [TABLE2] AS T2 ON ...

OPTION (FORCE ORDER)

As with any sort of query hint, you should be extra careful that you are doing the right thing. For instance, the join order chosen for a query can change between multiple runs due to several reasons. The cardinality of the join tables could have changed significantly, or the data distribution could have changed. So even though the join order that you selected at the time of deployment was optimal, it may not remain so all the time. Ordinarily the optimizer would decide the join order on fly and therefore can adjust to such changes.

Not so when the join order has been forced.

That's all for the time being.

-Pragya Agarwal

Dropping Defaults (Column Constraint)

2008-04-03T18:13:00Z

There were some recent changes that were done in regards to column defaults. In this blog entry I want to bring out, what used to happen and what happens now.

I have discussed the case of dropping a default on a column in this post, however, in a case where you are facing problems dealing with column specific constraints(especially dropping), the appraoch I suggest below should help you with that.

What used to happen in version 3.1 and earlier?

Consider the following query:

CREATE TABLE foo (col INT DEFAULT 1, col2 INT CONSTRAINT cons DEFAULT 2);

Usually when defining defaults one would choose the way it has been done for first column “col”, but even the other options as shown for second column “col2” also works.
In version 3.1 and earlier, if at all you ever decide that you don’t want the default on “col2”, you could issue a statement like this:

ALTER TABLE foo DROP CONSTRAINT cons;

Here, the default for “col2” as declared above will be stored as a constraint with name “cons”, so you could issue an ALTER statement on the table and drop it. It worked.

CREATE TABLE foo (col INT DEFAULT 1, col2 INT CONSTRAINT cons DEFAULT 2)
INSERT INTO foo (col) VALUES (0)
INSERT INTO foo (col2) VALUES (0)
INSERT INTO foo VALUES (0, 0)
INSERT INTO foo VALUES (default, default)
ALTER TABLE foo DROP CONSTRAINT cons
INSERT INTO foo (col) VALUES (0)
SELECT * FROM foo
                 col                col2
----------------------------------------
                   0                   2
                   1                   0
                   0                   0
                   1                   2
                   0              (NULL)

What changed in version 3.5?

In version 3.5 and onwards the constraint as declared above will not be table constraint anymore, rather the default constraint on a column will be maintained as column property at column level.

So, the statement,

ALTER TABLE foo DROP CONSTRAINT cons

would NOT work anymore. The “CONSTRAINT cons” from the original statement will be ignored. On issuing above mentioned ALTER statement, you will get an error saying its a bad constraint ID and it cannot be found.

To drop such constraint you will have to issue the following statement (this works on all versions and is the right way to drop the defaults than going for constraint names):

ALTER TABLE foo ALTER COLUMN col2 DROP DEFAULT

To see the column properties, you can query information_schema.columns. Look for the row with your column name, there will be two columns (COLUMN_HASDEFAULT and COLUMN_DEFAULT), they indicate whether you have the default on that column or not and the default value for that column, if any.

If you are feeling lazy to type in this, you can also do this through GUI, by editing the table schema in Visual Studio or SQL Server. In the table schema, go to the particular column and for the default property delete the value which you had previously entered. This is equivalent to issuing the above ALTER TABLE statement, in fact, behind the scenes they both use the same statement.

Ravi Tandon
SDET, SQL Server Compact

My First Entity Application Against SQLCE

2008-03-07T19:36:00Z

Now that SQL Server Compact 3.5 SP1 Beta is released it's time to take the first step, making your first entity application using SQLCE as a backend.

In his previous post, Ambrish gave steps on how to install SQL Server Compact SP1 Beta, along with Ado.Net Entity Framework Beta 3 and writing a small program to query the Northwind Database via LINQ.

In this post, I will briefly walk you through what Entity Framework Beta 3 has offer to, and how can we leverage it to write different types of entity queries against SQL Server Compact.

Going by EF Definition at: http://msdn2.microsoft.com/en-us/library/bb399572.aspx

The most unique selling point in my opinion becomes the part where you can run the same program/application against any backend (assuming the database schema is same), you have to just modify connection strings. Let's say even if the schema is little different, you can get that easily fixed in your metadata files, and boom, your app works.

I am including few screenshots which will guide you on how to go by making your own application and the complete program.

Please do read the previous post (by Ambrish) carefully as I will assume that you are done with the setup and we are ready to go. For those who could get things set up I am posting some screen shots to help you with that.

First step is to include an Ado.Net Entity Data Model into your project, give it a name that makes sense, I gave it SSCNorthwind.

The screenshot below, shows the database file that we selected, its connection string that will go into your App.config file and also the name you want to give it. For simplicity I chose NorthwindEntities.

The screenshot below shows you how to include the database file that you placed in your project root for generating data model.

The wizard below allows you to choose the tables that you want to be part of entity set, as SQL CE does not support stored procedures and views as of now, we will ignore those two options.

The red box in below shows the change that we have to make in App.Config to make this working J i.e. renaming the provider:

provider=Microsoft.SqlServerCe.Client.3.5 to provider=System.Data.SqlServerCe.3.5

This is being fixed, so in RTM version you won't have to do this manual step.

You can see that the name of Entity set is NortwindEntities, so whenever we make connection to this set, the connection string provided in App.Config file will be used.

Before:

After:

This is how the solution explorer should look now:

If you face any error while adding Northwind.sdf file into your project, just ignore it for now.

I have tried to include some comments in my program below to help you understand what's going on.

Points to notice:

To connect to a Database using an Entity Connection, you can do it in two ways.

First way is to specify your own connection string, like I have done at the start of program. Using that you can directly create an Entity Connection and work in the same way you used to work with SqlCeConnection and SqlCeCommand. That way can be seen in ESQL Query region.
The second approach is using Contexts or what we call here NorthwindEntities, (remember at top we renamed Entities to this for better understanding). For using context, you can do that in multiple ways

First, you can give a connection string and it will create the set.
Second, you can also first create an Entity Connection like above and then pass it on the NorthwindEntities, which is what I have done in Object Query section.
And the last approach for which we created App.Config in first place. You don't specify anything at all, the entity set (NorthwindEntities) picks up your connection string from App.Config and creates a connection. LINQ to Entites Query section is done using this approach.

Program (Contents of Program.cs):

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using System.Data.EntityClient;

using NorthwindModel;

using System.Data;

namespace MyFirstEntityApplicationForSSC

{

class Program

{

#region Static Declarations

//SQLCE Connection string for the sample database Northwind.sdf

static string _sqlCeConnnectionString = @"Data Source = Northwind.sdf";

//entityConnectionString: If your metadata(CSDL, SSDL and MSL, or in this case EDMX) files lie in the same place as application, you can use "."

static string _entityConnectionString =

String.Format(@"metadata=.;Provider=System.Data.SqlServerCe.3.5;provider connection string=""{0}""", _sqlCeConnnectionString);

//New Values that you might want to use for changing the data

static string _newFirstName = "AGENT";

static string _newLastName = "SMITH";

static string _newCountry = "MATRIX";

#endregion Static Declarations

#region Main

/// <summary>

/// MAIN: It calls the three different functions for following type of queries

/// Description in layman language

///

/// ESQL - This takes you back to old .Net 2.0 days where you used to set T-SQL statements and execute them, just that it here works on entities now instead of tables

/// Example Demonstrates: Simple Select Query using Entity SQL.

///

/// Object Query - Working on Data Objects was never so easy. This type of queries give you the power to work on data objects and most of all intellisense helps with things you can do

/// Example Demonstrates: Query and Updation of data and then Re-Querying

///

/// LINQ to Enity - LINQ means Language Integrated Query. This is a new and powerful way of writing queries that are checked for syntax at compile time itself. So, chances of failing in query logic at runtime become minimal

/// Example Demostrates: Query the data using default settings for connection in App.Config that was configured via EF Designer.

/// </summary>

/// <param name="args"></param>

static void Main(string[] args)

{

Console.WriteLine("SQL CE Northwind Entity App:");

try

{

EntityConnection entityConnection = new EntityConnection(_entityConnectionString); //Creating a new entity connection using the entity connection string

entityConnection.Open();

Console.WriteLine("\nESQL Query:");

ESQL_Query(entityConnection); //This does not use context, it's a simple esql command execution

Console.WriteLine("\nLINQ To Entity Query:");

LINQ_To_Entity_Query(); //This will make a new connection using the connection string in the App.Config file

Console.WriteLine("\nObject Query:");

ObjectQuery(entityConnection); //This will create the context with existing connection

entityConnection.Close();

}

catch (Exception ex)

{

Console.WriteLine("\nFAIL! Oops, it was not expected, an exception has been thrown, details below:\n");

Console.WriteLine(ex.Message);

}

Console.WriteLine("\nPress a Key to exit...");

Console.ReadKey();

}

#endregion Main

#region ESQL_Query

/// <summary>

/// ESQL_Query

/// (Executing a simple ESQL query against Northwind Entity Set)

/// Using the entity connection which is passed on as parameter

/// it executes a simple entity command on the Employees entity set in the Northwind Context.

/// The result of that query is later on printed on the screen using a data reader.

/// </summary>

/// <param name="entityConnection"></param>

private static void ESQL_Query(EntityConnection entityConnection)

{

EntityCommand entityCommand = entityConnection.CreateCommand();

entityCommand.CommandText = @"Select Emp.Employee_Id as Id, Emp.First_Name as Name from NorthwindEntities.Employees as Emp order by Emp.Employee_Id";

EntityDataReader entityDataReader = entityCommand.ExecuteReader(CommandBehavior.SequentialAccess);

//Note: You have to use CommandBehavior as SequentialAccess, otherwise an exception will be thrown

while (entityDataReader.Read())

{

for (int i = 0; i < entityDataReader.FieldCount; i++)

Console.Write(entityDataReader[i].ToString() + "\t");

Console.WriteLine();

}

#endregion ESQL_Query

#region LINQ_To_Entity_Query

/// <summary>

/// LINQ_To_Entity_Query

/// (Executing a simple LINQ query against Northwind Entity Set)

/// This function creates a new Northwind Context based on the setting provided for it in the App.Config File

/// It executes a simple LINQ query on the Employees entity set in the Northwind Context.

/// The result of that query is later on printed on the screen using the IQueryable structure.

/// </summary>

private static void LINQ_To_Entity_Query()

{

NorthwindEntities nwind = new NorthwindEntities(); //Uses the settings for connection string in App.Config File

IQueryable<string> emps = from e in nwind.Employees where (e.Employee_ID%2)==0 select e.First_Name; //Even Number Employee Ids

foreach (string e in emps)

{

Console.WriteLine(e);

}

#endregion LINQ_To_Entity_Query

#region ObjectQuery

/// <summary>

/// ObjectQuery

/// (Executing a simple Object query against Northwind Entity Set and Updating the data)

/// This function creates a new Northwind Context based on the entity connection thas has been passed to it as parameter

/// It executes a simple Object query on the Employees entity set in the Northwind Context.

/// The result of that query is later on printed on the screen and some changes are made to it.

/// The changes are then saved using SaveChanges on Context and then re-querying them to see if things went fine.

/// </summary>

/// <param name="entityConnection"></param>

private static void ObjectQuery(EntityConnection entityConnection)

{

NorthwindEntities nwind = new NorthwindEntities(entityConnection);

var emps = nwind.Employees.Where((delegate(Employees e)

{

return e.Employee_ID > 5;

}

));

//var emps = nwind.Employees.Where("it.Employee_ID > 5"); //This query is also equivalent to above one and will give same results

//var emps = nwind.Employees.Where("it.Employee_ID > @empid",new ObjectParameter("@empid",5)); //This query is also equivalent to above one, but it uses parameters for the values

foreach(Employees e in emps)

{

Console.WriteLine(e.Employee_ID + "\t" + e.First_Name + "\t" + e.Last_Name + "\t" + e.Country);

if (e.Employee_ID == 10) //As soon as we encounter employee id as 10, we want to change some data

{

Console.WriteLine("Changing Data");

e.First_Name = _newFirstName;

e.Last_Name = _newLastName;

e.Country = _newCountry;

}

Console.WriteLine("Saving Data Changes");

nwind.SaveChanges(); //Any changes made to the context are saved using this command

emps = nwind.Employees; //This is simplest way to select all employees. If you happen to choose particular data you may modify the query accordingly, like nwind.Employees.Where("it.Employee_ID = 10")

Console.WriteLine("Display Data Again:");

foreach (Employees e in emps)

{

Console.WriteLine(e.Employee_ID + "\t" + e.First_Name + "\t" + e.Last_Name + "\t" + e.Country);

}

#endregion ObjectQuery

}

The output of the program would look like this:

SQL CE Northwind Entity App:

ESQL Query:

1 Nancy

2 Andrew

3 Janet

4 Margaret

5 Steven

6 Michael

7 Robert

8 Laura

9 Anne

10 Albert

11 Tim

12 Caroline

13 Justin

14 Xavier

15 Laurent

LINQ To Entity Query:

Andrew

Margaret

Michael

Laura

Albert

Caroline

Xavier

Object Query:

6 Michael Suyama UK

7 Robert King UK

8 Laura Callahan USA

9 Anne Dodsworth UK

10 Albert Hellstern USA

Changing Data

11 Tim Smith USA

12 Caroline Patterson USA

13 Justin Brid France

14 Xavier Martin France

15 Laurent Pereira France

Saving Data Changes

Display Data Again:

1 Nancy Davolio USA

2 Andrew Fuller USA

3 Janet Leverling USA

4 Margaret Peacock USA

5 Steven Buchanan UK

6 Michael Suyama UK

7 Robert King UK

8 Laura Callahan USA

9 Anne Dodsworth UK

10 AGENT SMITH MATRIX

11 Tim Smith USA

12 Caroline Patterson USA

13 Justin Brid France

14 Xavier Martin France

15 Laurent Pereira France

Press a Key to exit...

If you want your changes to persist or not to persist, modify the properties for the Northwind.sdf file in your solution properties as per your desire. By default it is "copy if newer" which means the database file in your execution folder will retain the changes. By changing it to "copy always" you can refresh your database each time.

I hope now you can now go back and start writing advanced programs using various techniques available.

Ado.Net Entity Framework is a new weapon to the list of a current developer's arsenal, opening multiple ways in which he can write his programs without worrying about the under lying intricacies.

The shift to entity plane seems very lucrative, easy and promising. And what more, all of this comes with handy tools which make life easy. Writing your code from the word Go!

The above topic was in relation to LINQ to Entities, if you are looking for LINQ to SQL refer to this blog post: http://blogs.msdn.com/sqlservercompact/archive/2007/08/21/linq-with-sql-server-compact-a-ka-dlinq-over-sql-ce.aspx

Happy Programming!

Ravi Tandon

SDET, SQL Server Compact

Download available for SQL Server Compact 3.5 SP1 Beta for ADO.Net Entity Framework Beta 3

2008-02-21T09:30:00Z

The SQL Server Compact 3.5 SP1 Beta for ADO.Net Entity Framework Beta 3 is available for download at Microsoft Download Center.

To install SQL Server Compact 3.5 SP1 Beta Release for ADO.NET Entity Framework Beta 3

Install ADO.NET Entity Framework Beta 3 from Microsoft Download Center. You must ensure that the pre-requisites specified on the download page are installed before installing ADO.NET Entity Framework Beta 3.
In Control Panel, double-click Add or Remove Programs. Uninstall SQL Server Compact 3.5 or SQL Server Compact 3.5 SP1 Beta.
Install SQL Server Compact 3.5 SP1 Beta Release for the ADO.NET Entity Framework Beta 3 (SSCERuntime-ENU.msi) from the Microsoft Download Center. Note that administrator rights are required on a computer to install SQL Server Compact 3.5 SP1 Beta Release for ADO.NET Entity Framework Beta 3.

To install ADO.NET Entity Framework Tools December 07 Community Technology Preview (CTP)

Install the ADO.NET Entity Framework Beta 3 and SQL Server Compact 3.5 SP1 Beta Release for the ADO.Net Entity Framework Beta 3 as specified above. Install ADO.NET Entity Framework Tools December 2007 CTP from the Microsoft Download Center. You must ensure that the pre-requisites specified on the download page are installed before installing ADO.NET Entity Framework Tools December 2007 CTP. The CTP installs on a Visual Studio 2008 installation.
Copy the SQL Server Compact 3.5 SP1 Design Tools Beta (SSCEVSTools-ENU.exe) to your computer by clicking on Save or Save this program to disk and after the download is complete install SSCEVSTools-ENU.exe by double-clicking it. Do not install the SSCEVSTools-ENU.exe by clicking on the Run. While running the SSCEVSTools-ENU.exe file, Microsoft.SqlServerCe.Client.dll is extracted to %ProgramFiles%\Microsoft Visual Studio 9.0\Common7\IDE folder. The SSCEVSTools-ENU.exe will create a backup of the existing DLL at the location %ProgramFiles%\Microsoft Visual Studio 9.0\Common7\IDE.

For uninstalling SQL Server Compact 3.5 SP1 Beta refer to the read me.

The steps to create a sample application are as given below:

1. Start Visual Studio 2008 and create a console project by clicking on the menu item File à New à Project. In the New Project window expand Visual Basic or Visual C#, select Windows and then select Console Application. For the project the .Net Framework version should be .Net Framework 3.5. Name the project as SQLCompactEDMProject. On clicking OK the project is created

2. To generate the Entity Data Model for the Northwind.sdf, copy the Northwind.sdf from the folder %Program Files%\Microsoft SQL Server Compact Edition\v3.5\Samples to the users folder like C:\Users\<login name>\Documents\Visual Studio 2008\Projects\SQLCompactEDMProject\SQLCompactEDMProject. Select the SQLCompactEDMProject in the Solution Explorer and right-click, point to Add, and then click New Item. Select ADO.NET Entity Data Model in the Templates pane. Enter Northwind.edmx for the model name and click Add. The opening page of the Entity Data Model Wizard is displayed.

3. Select Generate from database in the Choose Model Contents dialog box and click Next. Click the New Connection button. The Connection Properties dialog box is displayed.

4. In the Connection Properties dialog click the button Change in the Data Source. In the Change Data Source dialog box select Microsoft SQL Server Compact 3.5 and click OK. Click the radio button My Computer in the Data Source. In the Connection Properties browse to the Northwind.sdf and click OK in the Connection Properties dialog. The Choose Your Data Connection screen in the Entity Data Model Wizard comes up.

5. The checkbox in front of Save entity connection settings in App.Config as: will be clicked by default. Change the name in the dialog from Entities to NorthwindEntities. Click Next to continue.

6. The Choose Your Database Objects dialog box is displayed. By default all the tables in the database are selected and will be included in the EDM. Uncheck the box next to Tables. This deselects all the tables in the database. Expand the Tables node and select the Customers, Orders, and Products tables. Change the Model Namespace: from Model to NorthwindModel

7. Click Finish to complete the wizard.

a. The wizard does the following:

b. Adds references to the System.Data, System.Data.Entity, System.Core, System.Security, and System.Runtime.Serialization assemblies.

c. Generates an .edmx file which encapsulates the information from the EDM mapping files.

d. Creates an App.Config file.

8. Double click the App.Config file in the Solution Explorer to open it. Change the provider entry from provider=Microsoft.SqlServerCe.Client.3.5 to provider=System.Data.SqlServerCe.3.5

9. Add an Imports (Visual Basic) or using (C#) statement at the beginning of the file to reference the EDM created from the Northwind database. (The name of the model corresponds to the value of the namespace attribute in your .edmx file).

Visual Basic

Imports SQLCompactEDMProject.NorthwindModel

using NorthwindModel;

10. Add the following code to the Main subroutine (Module1.vb or Program.cs)

Visual Basic

Using db As NorthwindEntities = New NorthwindEntities

Dim customers = From c In db.Customers _

Where c.City = "London" _

Order By c.Company_Name Select c

For Each c As Customers In customers

Console.WriteLine(c.Company_Name)

Console.ReadLine

End Using

using (NorthwindEntities db = new NorthwindEntities())

{

var customers = from c in db.Customers

where c.City == "London"

orderby c.Company_Name

select c;

foreach (Customers c in customers)

{

Console.WriteLine(c.Company_Name);

}

Console.ReadLine ();

}

11. Press CTRL+F5 to run the application. After the results display on the console screen press enter to exit the application

For more information and samples please see the ADO.NET Entity Framework Beta 3 Documentation and ADO.NET Entity Framework Samples

Regards,

Ambrish Mishra

Program Manager - SQL Server Compact