Welcome to MSDN Blogs Sign in | Join | Help

CRM 4: System-wide Duplicate Detection

Here I am continuing the duplicate detection story of CRM 4.0 with a new aspect of this feature. Organizations face a common problem where in spite of several policies and guidelines in place users end up entering duplicate data in MSCRM system via data import or data generated during many internal operations or simply knowingly creating duplicate records. Creating duplicate records at any point of time might be crucial but this ends in a lot of duplicate data in system which can affect sales, marketing and auditing of the organization. System-Wide Duplicate detection feature in Titan solves this problem and empower users to find duplicates in a part or all the records of an entity in the MSCRM system.

System-wide duplicate detection is an asynchronous type job and it detects all the duplicates in the system for the given records. System-wide duplicate detection can be run on selected records of any entity or records under any specific user query or all the records of that entity. I will cover each of these scenarios one by one.

System-wide duplicate detection wizard can be launched from Duplicate Detection option under Tools menu in MSCRM or Workplace > Duplicate Detection grid New button. First page of this wizard is welcome page and informs user about the capabilities of this wizard.

clip_image002

Proceed to Select Records page to select the entity on whose records you are going to run the duplicate detection. All the entities that have one or more duplicate rules published in MSCRM system will be listed under Look for dropdown on this page. This interface is standard Advance find interface and you can either select any existing view to run the duplicate detection on all the records of that view or can create a custom query also. You can also see the records identified according to the selected query by clicking Preview Records button.

clip_image004

Now Proceed to Select Options page for giving name to your job and selecting the time at which you want to run this job. This duplicate detection job can be run recursively also by checking “Run this job after every” check box and providing number of days in the days drop down box. This feature is very useful in scenarios where duplicates are creating regularly in MSCRM system and you want to keep track of them after every few days or months. You can also choose to send an email to you and others on completion of this job.

clip_image006

Now go next to Start Duplicate detection Job page and submit the job. Your job will be submitted for processing at the time provided by you in the wizard.

Duplicate Detection can also be launched from any entity grid for which one or more duplicate rules are published. While running duplicate detection on entity grid you can choose either to run duplicate detection for manually selected records or for all records under that view. For running duplicate detection for selected records first select one or multiple records on grid and go to grid view of desired entity and launch duplicate detection dialog from More Actions > Detect Duplicates > For Selected Records. This will launch Check for Duplicates dialog which is same as Select Options page of System Wide duplicate detection wizard. Just fill your options and your job will be submitted for processing.

clip_image008

clip_image010

For running duplicate detection for all records in any view select More Actions > Detect Duplicates > For All Records on All Pages.

You can see your duplicate detection jobs under Workplace > Duplicate Detection grid or Settings > Data Management > Duplicate Detection Jobs grid. Here you can check the status of your jobs and see the duplicates found in any duplicate detection job. Open the system form by double clicking and go to View Duplicates under Details section on left hand side.

clip_image012

This form gives user the power of editing, merging and deleting the duplicate records which is essential for data cleansing.

Some noticeable points about System wide Duplicate detection are

a) Duplicate Detection runs for all the published rules for any entity and there is no concept of choosing duplicate rules while doing duplicate detection.

b) Duplicate detection wizard can be run on only one entity at a time. Though you can run multiple jobs for different entities.

c) There should be duplicate rules present as published for the entity you want to run duplicate detection.

d) User level privileges apply while showing duplicate records to users.

This story will continue and in next post I will discuss security model of duplicate detection.

Sudhakar Singh

Published Thursday, January 17, 2008 9:02 AM by crmblog

Comment Notification

If you would like to receive an email when updates are made to this post, please register here

Subscribe to this post's comments using RSS

Comments

# » CRM 4: System-wide Duplicate Detection Microsoft CRM Development Information: Information and Research on Microsoft CRM Development

# re: CRM 4: System-wide Duplicate Detection

Sudhakar,

Very helpful disucssion on dup detection.  Question - is dup detection run automatically when Outlook is synchrnized with CRM? For example,a new contact is entered offline, when returning online is the contact checked for dup detection?

Friday, January 18, 2008 10:34 AM by dmbrandon

# Duplicate Detection while creating/updating records in Titan

Welcome to the next in the series of posts on Duplicate Detection in Titan. In case you missed the action

Monday, January 21, 2008 1:14 PM by Microsoft Dynamics CRM Team Blog

# re: CRM 4: System-wide Duplicate Detection

Hi dmbrandon,

Duplicate detection runs automatically when Outlook goes online from offline if settings says so. By default it is disabled. To enable it in outlook go to CRM > Options menu > Synchronization tab and select Do not create duplicates option and now when going from offline to online outlook will throw Data Synchronization error dialog if any duplicate is found and will ask you to choose appropriate actions like Resolve duplicate or Ignore duplicate.

Monday, January 21, 2008 11:37 PM by Sudhakar

# re: CRM 4: System-wide Duplicate Detection

I am imporeting data via Data migration manager and not through Bulk import wizard , can dupicate detection work while inserting data thro DMM tool ?

Wednesday, February 06, 2008 9:56 AM by Mohit

# re: CRM 4: System-wide Duplicate Detection

No, Duplicate detection doesn't work with DMM. This is as per design so that user dont get a lot of errors during migration. To clean the data after migration you can anytime run the bulk duplicate detection wizard.

Thursday, February 07, 2008 1:45 AM by Sudhakar

# re: CRM 4: System-wide Duplicate Detection

Excellent article!

Quick question - In my environment, then when selecting 'send email to me when job is finished', the email message has a link using an internal name of the crm server and NOT the fully qualified name that is used to access the system.

The fully qualified name with https is displayed OK when sending 'short cuts' from within the crm system.

Any ideas on where I may be able to fix this?

Thanks a bunch!

Monday, February 11, 2008 3:41 PM by Alex F

# re: CRM 4: System-wide Duplicate Detection

This is currently a limitation where the hyperlink works fine for on premise deployments. This will be fixed in next version.

In single organization environment, you might give a try to replacing '@servername' with  fully qualified name in the following email templates:

1. Bulk Deletion Task Completed With Failure Template

2. Bulk Deletion Task Completion Template

3. Bulk Deletion Task Failed Template

I should warn you that in multi organization environment there might not be a workaround which works for all.

Wednesday, March 12, 2008 1:59 AM by Sharad

# re: CRM 4: System-wide Duplicate Detection

Hi Alex,

You can update the URL in the e-mail template being used for duplicate detection completion.

For this navigate to

Settings->Templates->E-mail Templates and open

"Duplicate Detection Task Completion Template".Here you can update the link to reflect the fully qualified name.Once done all e-mails specific to this template will be created with updated link.

Wednesday, March 12, 2008 7:01 AM by Sonal

# re: CRM 4: System-wide Duplicate Detection

The detection works very fine!

How is it possible to delete the detected records?

Thanks,

Bernhard

Thursday, March 27, 2008 2:01 PM by Bernhard

# re: CRM 4: System-wide Duplicate Detection

Hi Bernhard,

In the View Duplicates dialog itself, on lower duplicate records grid toolbar there are options for deleting/deactivating/editing/merging the duplicate record.

Monday, April 14, 2008 6:46 AM by Sudhakar

# Bulk Record Deletion in Microsoft CRM Online

This blog post will cover custom filter-based mass record deletion for a given entity System. Note: If

Thursday, April 24, 2008 11:17 AM by Microsoft Dynamics CRM Team Blog

# Bulk Record Deletion in Microsoft CRM Online

This blog post will cover custom filter-based mass record deletion for a given entity System. In this

Thursday, April 24, 2008 11:59 AM by Microsoft CRM Live Team

# Bulk Record Deletion in Microsoft CRM Online

This blog post will cover custom filter-based mass record deletion for a given entity System. In this

Thursday, April 24, 2008 12:14 PM by Microsoft Dynamics CRM Team Blog

# Duplicate Detection Security Model

Security model of duplicate detection is in accordance with the security flow of MSCRM v4 which is very

Tuesday, April 29, 2008 11:33 AM by Microsoft Dynamics CRM Team Blog

# Duplicate Detection Security Model

Security model of duplicate detection is in accordance with the security flow of MSCRM v4 which is very

Tuesday, April 29, 2008 12:09 PM by Microsoft CRM Live Team

# Duplicate Detection Security Model

Security model of duplicate detection is in accordance with the security flow of MSCRM v4 which is very

Tuesday, April 29, 2008 12:26 PM by Microsoft Dynamics CRM Team Blog

# re: CRM 4: System-wide Duplicate Detection

I started a duplicate detection rule and now I cannot stop it.  I get an error message "The duplicate detection rule is currently being used and cannot be updated or deleted. Please try again later."  

I cannot delete, unpublish, edit or anything else.  I created this for use with a custom entity and cannot delete the entity either.

Tuesday, July 01, 2008 9:59 AM by stacy

# re: CRM 4: System-wide Duplicate Detection

There may be running rule(s) that are in "wait" mode.  Drill down on the duplicate detecton rule via system jobs and find out which ones are pending/wait and cancel those.  You can then delete, unpublish, edit, etc.

Frank Lee, Microsoft Dynamics CRM MVP

http://microsoft-crm.spaces.live.com

http://www.workopia.com/Links.htm

Tuesday, July 01, 2008 3:27 PM by Frank Lee

# re: CRM 4: System-wide Duplicate Detection

I can not remove a rule on Contacts.  This started 7-21.  It has no rule in wait or pending mode.  Please help

Tuesday, July 22, 2008 1:29 PM by Chris Kiec

# re: CRM 4: System-wide Duplicate Detection

2 years ago someone added some products, quotes, orders and invoices to our production CRM system when testing this feature.  We now want to use the integrated ordering processes but want to remove the old test data - because they are related records (i.e. a product is contained in an order which was cancelled) I'm unable to remove the data through the GUI.  What other methods can I use to remove this redundant data?

Sunday, August 17, 2008 6:29 PM by Peter McKenzie-Bridle

# re: CRM 4: System-wide Duplicate Detection

Hi Im trying to remove duplicate Account names is there any way to do this, every time i select account it then gives me equals not equals etc and wants a value, thing is if i put in a value I would be as well going through manually, all I want it that any accounts with the same name are highlighted so I can remove them is this possible ?

Monday, September 08, 2008 5:54 AM by Kevin Roy

# re: CRM 4: System-wide Duplicate Detection

I too cannot unpublish a system rule either and there is nothing in a wait status either.  

Tuesday, September 30, 2008 5:42 PM by reicoskyr

# re: CRM 4: System-wide Duplicate Detection

I also cannot unpublish a rule, there is nothing in wait status. (there is nothing about duplicate detection in system jobs)  

I've deleted the results of the last duplicate detection job I ran, with no luck. Have logged out of CRM and logged back in and still unable to unpublish.

Tuesday, October 21, 2008 2:14 PM by Debbie Grale

# re: CRM 4: System-wide Duplicate Detection

Duplicate Detection runs for all the published rules for any entity and there is no concept of choosing duplicate rules while doing duplicate detection.

What does the above line mean?

Actually I created a workflow on Email creation and has nothing to do with the duplicate detection rule created for lead entity. But there seems to be an entry of a system job of this duplicate detedtion rule under that workflow created on the email entity!!

Any suggestions?? Please help!

Regards,

RC

Thursday, February 05, 2009 1:25 AM by RC@

# re: CRM 4: System-wide Duplicate Detection

I was also unable to unpublish/delete duplicate detection rule. I tried searching for the duplicate-detection rule in waiting state from Settings>>System jobs but i couldn't find the job in waiting state. Thanks to "Advance Find" which helped me to find the duplicate-detection job in waiting state. I used following query: Look For: System Jobs

System Job Type: Equals : Bulk Duplicate Detection

Status Reason : Equals : Waiting

This query displayed a job that was in waiting state. After cancelling the job I was able to unpublish the rule.

Tuesday, February 17, 2009 4:18 AM by P Uday Kumar

# re: CRM 4: System-wide Duplicate Detection

System Admin cannot publish the the rule. Publish button not seen

Friday, April 03, 2009 3:09 AM by Rafiq

# re: CRM 4: System-wide Duplicate Detection

Duplicate detection rules have been unpublished and re-published and each time they are stuck in 'Publishing' status.

Tried creating new rules after the re-deploy and get the same results when publishing these.

Any ideas about how I can get the rules to publish?

Friday, April 10, 2009 11:29 AM by Bhaskar

# re: CRM 4: System-wide Duplicate Detection

Hi,

does anyone know how to set a system wide dedupe rule that ignores deactivated accounts?  

thanks

Tuesday, April 21, 2009 5:38 AM by Sam

# Unhanded Exception; Error Code -2147220969

Duplicate detection has been failing since Feb. In the System Job Failure with Errors view, I get a Failed Status with an error code -2,147,220,969.

The Text is Unhanded Exception: System.Web.Services.Protocols.SoapException: Server was unable to process request. Detail:<detail><error><code>0x80040217</Code><descrition>systemuser With Id = cfc99cba-ca58-dd11-bc42-001ec9365ec7 Does Not Exists</description><type>Platform</type></error></details> atSystem.Web.Services.Protocols.SoapHttpClientProtocol.RealResponce(SoapClentMessage message, WebResponce responce, Stream responceStream, Boolean asyncCall) at System.WebServices.Protocols,SoapHttpClientProtocol.Invo

In the users window I can double click on each user and see the assisgned UserID. Non of my users ahve this ID. It must be an internal ID.

Little advice?

Thursday, May 21, 2009 2:49 PM by SJM

# re: CRM 4: System-wide Duplicate Detection

The deduplication tools in CRM 4.0 are fine but the deduplication rules are that basic that they are not a helpful feature of a state of the art CRM

Sunday, July 26, 2009 10:59 AM by Fred Kaemmer

# re: CRM 4: System-wide Duplicate Detection

I have the same problem with the publishing status of the duplicate rules. Does anyone find a solution. How can I resolve this issue?

Saturday, October 17, 2009 2:14 AM by Yiannis Patrikiadis

# re: CRM 4: System-wide Duplicate Detection

Is there a way to change the number of rows that display in the upper portion of the screen when viewing a duplicate detection job?  I can only see 12 rows at maximum.  Also, would be nice if there was an Export to Excel option for the list.

Monday, October 26, 2009 9:49 AM by Hansel Gillis

Leave a Comment

(required) 
required 
(required) 

  
Enter Code Here: Required
 
Page view tracker