The Microsoft Dynamics CRM Blog
News and views from the Microsoft Dynamics CRM Team

Duplicate Detection - First step towards maintaining Data Quality

Duplicate Detection - First step towards maintaining Data Quality

  • Comments 9

Today, maintaining data quality is one of the most critical problems faced in the business. Since data is entered in the system by different people, in different standards and at different levels, so this results in several data quality issues. One of the top most issues is having multiple representation of same logical real world entity or in other words having duplicate records. Such duplicate records may lead to significant problems to business for example:-

· Increased cost to company

· Increased customer dissatisfaction

· Erroneous data mining & analytics etc

In Titan there is a new very powerful feature of “Duplicate Detection” which will help organizations take first step towards maintaining quality of data in their system. So here is the brief introduction to this feature:-

· Duplicate detection feature in Titan, will allow organizations to define duplicate detection policy (Duplicate detection rules) for different record types (supported for all record types including custom entity). Using intuitive and well known (advanced find like) user interface, an administrator can define duplicate detection policies based on which system should report a record as a duplicate of another existing record. These duplicate detection policies can be defined across different record types for example organization may define that a lead will be duplicate if a contact is already existing with same name and phone number. Necessary knobs are exposed using which organization can turn on / off duplicate detection for different record types and different data entry points.

· Based on the duplicate detection policy (rules) defined by the administrator, system will alert user about the potential duplicates when user tries to create new or update existing record. Same policy will be applicable for other data entry points as well for example data import, Offline to online data sync, Promote contacts in outlook, SDK etc. So in short all the entry points for data are guarded with duplicate detection to just allow high quality data into the system.

· Even if the entry points are guarded with duplicate detection, there could be some legitimate business reasons due to which duplicate data enters the system. So to help organization maintain data quality on the on-going basis we have introduced a notion of “System-Wide duplicate detection”. With this feature you can schedule a duplicate detection background job which will check for duplicates for all the records matching a criterion (Advanced find query) and report back the result (list of duplicates). User can cleanse the data by deleting, deactivating or merging the duplicates reported by background job.

This is just a overview; expect more details on each and every concept of this super cool feature in much more details soon.

This posting is provided "AS IS" with no warranties, and confers no rights.

Rohit Bhatia

  • PingBack from

  • Really a cool feature that all CRM geeks are expecting in Titan. We used to do a lot in CRM 3.0 in this regard. Thanks MS for this cool new feature.

  • Hi Rohit

    Great Post !!

     When c360 has duplicate detaction product already in the market. And C360 is a big partner of Microsoft why again Microsoft has come with this product!

    Dont you think this will make other partners worry about  devloping new application or Addons to MSCRM.


    Rakesh Agarwal

  • I have tried this, and it's good feature, we implement a lot of customization for this feature, and now it's available out of the box, which is cool.

    But I notice in CTP version that this Data Duplication is working fine when I do "System-Wide check". But, when I create/update the record, no alert is triggered at all. The record is just simply created and saved.

    I have suggestion that the alert can be formed as a warning to the user, which user can choose to save it anyway, or validation, that user cannot save it at all. This is common requirement for most of our CRM implementation. Some customer allows to save the data as long as, the user has been alerted. But others want it to be restricted. And since, there is "System-Wide check", there is always a way to validate back. :)

  • Hi Rakesh,

    Based on our customers feedback we came to know that most of our customers considered duplicate detection as one of the most fundamental feature and expected it to be available out-of-box. So to have better customer satisfaction we decided to have this feature in Titan. However while building this feature we were cognizant of the fact that there are strong Microsoft partners already offering this feature, so we decided to build this feature extensible enough so that our partners can build more focused & specific solutions for our customers on top of it. We plan to have some posts which will demonstrate power of duplicate detection platform which can be used to build stronger solutions.

    So I see it as Win-win situation for our customers as well as partners.


    Rohit Bhatia

    This posting is provided "AS IS" with no warranties, and confers no rights.

  • Hi Gunandy,

    Duplicate detection during create and update of record is available. Due to some know issue this didnt work in CTP. This will be working exactly the way you expected by Titan release.


    Rohit Bhatia

    This posting is provided "AS IS" with no warranties, and confers no rights.

  • Does anyone have recommendations for what the rules should be?  There are many possibilities but which ones work the best particularly when the new record may be coming in over the web where you don't really want to ask someone if they are one of some existing list?

  • In the previous post on the Duplicate Detection Rohit described the power of this feature. In order to

  • Welcome to the next in the series of posts on Duplicate Detection in Titan. In case you missed the action

Page 1 of 1 (9 items)
Leave a Comment
  • Please add 7 and 8 and type the answer here:
  • Post