The Microsoft Dynamics CRM Blog
News and views from the Microsoft Dynamics CRM Team

Microsoft Dynamics CRM 4.0 Data Migration Performance

Microsoft Dynamics CRM 4.0 Data Migration Performance

  • Comments 7

This blog post is intended to provide necessary information regarding performance of the Data Migration feature. I will provide some details on the tests and results on a specific hardware environment and some guidance on the deployment side when performing a high volume data migration.

Hardware Configuration:

I have used this hardware set for the data migration tests covered in this blog.

  • Domain Controller : Dell Power Edge 2850, 2 Proc Xeon 3.6 GHz, 2 MB L2 Cache, 800 MHz FSB, 4 GB DDR2 RAM
  • Application Server : Dell Power Edge 2850, 2 Proc Xeon 3.6 GHz, 2 MB L2 Cache, 800 MHz FSB, 4 GB DDR2 RAM
  • SQL Server : Dell Power Edge 2850, 2 Proc Xeon 3.6 GHz, 2 MB L2 Cache, 800 MHz FSB, 4 GB DDR2 RAM
  • DMM Machine : HP Compaq dc 7700 SFF Intel Core 2 Duo 2 GHz, 2 MB L2 Cache, 800 MHz FSB, 2 GB DDR2 RAM

Software Configuration: The domain controller, application server and the SQL server were running Windows 2003 Enterprise server sp2. The SQL server was configured to run an instance of SQL server 2005 Enterprise edition sp2. The DMM machine was prepared with Windows XP professional edition sp2 and was running an instance of SQL Server 2005 Standard edition sp2. All the latest hotfixes and updates were applied in the machines.

Workload: The Migration scenario is valid for a fresh installation. The system was having one organization with 200 users. One user (with system administrator role) was dedicated for data migration. There was no static data load or run time load of users on the test system when migration was in progress.

System Settings: No workflow was defined during the course of migration. Duplicate detection was turned off.

Scenarios:

Based on different size and need of the organization, we came up with the following three migration scenarios.

  1. Simple: An organization migrates Leads, Contacts, Campaigns and similar entities without any relationship. This is the simplest form of data and does not involve any complex mapping or lookup resolution. DMM intelligently migrates this type of independent data in a parallel fashion and achieves very high throughput.
  2. Moderate: An organization migrates records with picklist and lookup fields defined. But there is no complex transformation defined.
  3. Complex: An organization using some other CRM, migrates to Microsoft CRM. There are multiple entities with rich interdepending relationships and large number of records. The used map involves picklist, lookup and complex transformations.
  4. All these three scenarios are unique in nature and the migration time can be very different with these situations. I am listing the scenarios and the end to end throughput (records per second) that I got while executing these scenarios.

Scenario Complexity

Data Size

Migration Time

End to end throughput (records per second)

Simple (Without any picklist, lookup or complex transformations)

5 million

20 hr

70

Moderate (picklist and lookup transformation but no complex transformation)

1 million

9.2 hr

30

Complex (14 CRM entities with picklist, lookup and complex transformation)

2.5 million

37 hr

19

Let me elaborate more on the complex migration scenario. The migration data was very high volume and spanning over 14 CRM entities with SF like data. The entity wide record distribution is provided in the below table.

Entity

Count

Entity

Count

Account

100000

Opportunity

500000

Contact

200000

OpportunityProduct

500000

Lead

500000

Product

10000

Contract

200000

PriceLevel

10000

Case

100000

ProductPriceLevel

10000

Event

100000

Solution

10000

Task

200000

Campaign

10000

The file sizes for some of the entities were more than 32 MB and hence I had to split them and adjust the out of box SF map. The resultant map was mostly same as the original map, only I need to duplicate the mapping for the entities where I had split the data files.

The above tests show that DMM can handle data of different complexity and can migrate really high volume of data. The migration time and end to end throughput actually varies a lot depending on the complexity of your data. But these results will give you a starting point if you wish to know how much time the migration may take in your deployment.

Tips for high volume Data Migration:

Before you do a migration involving huge number of records, I suggest you to do the following steps.

  1. Deploy your CRM server according to your business need and targeted capacity. DMM should be installed in a 2 proc 32 bit machine with at least 4 GB RAM.
  2. First do a test migration with very lees records but the same map and same type of data. This will help you to know whether the map is appropriate and you can undo the test migration, by using “Delete Data” option in DMM.
  3. If you do not want to automatically customize the CRM server during migration, then uncheck that option during migration. The repercussion is that custom attributes of picklist types will not be populated automatically.
  4. Do not migrate data using a virtual machine. Use a physical high end machine to achieve faster migration.
  5. Follow the below mentioned optimization techniques, wherever possible.

Optimization Techniques:

  1. All the machines in the system must be in the same time zone and use same regional settings.
  2. In the Internet Options -> Connections -> LAN Settings, set the physically nearest proxy server and set ‘Bypass proxy server for local addresses’. Do not use the option ‘Automatic Configuration’ as it involves in IL code generation (observed for .NET 2.0 SOAP classes).
  3. For best results, make sure that the DMM and CRM server are on a fast LAN connection with no TCP routing hops or HTTP proxies in between.
  4. Dynamics CRM 4.0 application has been launched at least once before testing.
  5. Make sure that tracing is off in both server and DMM. Tracing might drastically slowdown the application.
  6. For best results, CRM server, SQL server and DMM should be in separate boxes.
  7. If user has a license for SQL Server 2005, then it is advisable to have DMM database in that SQL server and not in SQL Server 2005 Express edition. SQL Express is not advisable if you intend to do a high volume data migration.
  8. Follow the optimization techniques mentioned in the whitepaper Optimizing and Maintaining Microsoft Dynamics CRM 4.0.

Conclusion:

The above mentioned results outline certain aspects of Data Migration performance for Dynamics CRM 4.0. The combination of fast algorithms used in Microsoft Dynamics CRM 4.0 Data Migration feature, along with the new generation Intel Xeon servers results in the high throughput for complex Asynchronous Data Migration jobs.

When reading this blog, user should keep in mind the following points.

  • The above results are obtained when only one user is performing Data Migration. The results may not be same if multiple users are performing data migration or other operations are being executed concurrently.
  • This test was done in an on premise CRM deployment and the results may not be same for CRM online.
  • There is no workflow or plug-in defined during the course of migration.

Any deployment in a similar environment may not provide the exact same numbers, but the provided numbers indicate the pattern of performance numbers for Data Migration. This can be useful in initial planning for migrating your data into CRM using Data Migration Manager.

Koushik Bhattacharjee

  • PingBack from http://www.travel-hilarity.com/airline_travel/?p=4926

  • This blog post is intended to provide necessary information regarding performance of the Data Migration feature. I will provide some details on the tests and results on a specific hardware environment and some guidance on the deployment side when performin

  • How did you configure the mapping xml for the Complex Scenario?

    Greets

  • For the complex scenario, I started with the Out of Box SF map. For some of the entities, like lead, opportunity, the file size was more than 32 MB. So I split them into multiple files like Lead1,Lead2. Then I modified the xml map in such a way that the mapping section of Lead is now copied two times. One is mapping for Lead1, another for Lead2 (but the mapping information is same as before). I saved the modified map and used that for migration.

  • Hi,

    You state that the machine for running DMM should not be a VM, in our environment we are running with an XP VM on 2008 with Hyper-V. Is this likely to still slow things down (given that hyper-v allows more direct hardware access)?

    Thanks,

    Ben.

  • Hi Ben,

    We did not perform migration in Hyper - V machines. But looking into some other test result executed on Hyper - V and VPC, we found that Hyper - V provides much more speed than VPC as you mentioned. Still it may not be exactly same as a dedicated physical machine. But if VPC and Hyper - V are the only alternatives, Hyper - V will be better. And if your base hardware running Hyper - V is a powerful server (HP DL series or equivalent) and you have provided sufficient memory for the guest OS (here XP), and followed other optimization tricks, migration should not be slow.

    Thanks,

    Koushik

  • Hi,

    You can use SQL Server Integration Services (SSIS) to load data into the CRM. I believe this is the best way of doing that.

    I made a tool that helps generates the DLL required to use from SSIS script component to write Microsoft SDK code. You can download this too for free from this location http://www.ssis4crm.com

    You will fined there intsractions of doing a simple load.

    This way is supported by Microsoft (using MSCRM SDK)

Page 1 of 1 (7 items)
Leave a Comment
  • Please add 2 and 3 and type the answer here:
  • Post