This blog post is intended to provide necessary information regarding performance of the Data Migration feature. I will provide some details on the tests and results on a specific hardware environment and some guidance on the deployment side when performing a high volume data migration.
I have used this hardware set for the data migration tests covered in this blog.
Software Configuration: The domain controller, application server and the SQL server were running Windows 2003 Enterprise server sp2. The SQL server was configured to run an instance of SQL server 2005 Enterprise edition sp2. The DMM machine was prepared with Windows XP professional edition sp2 and was running an instance of SQL Server 2005 Standard edition sp2. All the latest hotfixes and updates were applied in the machines.
Workload: The Migration scenario is valid for a fresh installation. The system was having one organization with 200 users. One user (with system administrator role) was dedicated for data migration. There was no static data load or run time load of users on the test system when migration was in progress.
System Settings: No workflow was defined during the course of migration. Duplicate detection was turned off.
Based on different size and need of the organization, we came up with the following three migration scenarios.
End to end throughput (records per second)
Simple (Without any picklist, lookup or complex transformations)
Moderate (picklist and lookup transformation but no complex transformation)
Complex (14 CRM entities with picklist, lookup and complex transformation)
Let me elaborate more on the complex migration scenario. The migration data was very high volume and spanning over 14 CRM entities with SF like data. The entity wide record distribution is provided in the below table.
The file sizes for some of the entities were more than 32 MB and hence I had to split them and adjust the out of box SF map. The resultant map was mostly same as the original map, only I need to duplicate the mapping for the entities where I had split the data files.
The above tests show that DMM can handle data of different complexity and can migrate really high volume of data. The migration time and end to end throughput actually varies a lot depending on the complexity of your data. But these results will give you a starting point if you wish to know how much time the migration may take in your deployment.
Tips for high volume Data Migration:
Before you do a migration involving huge number of records, I suggest you to do the following steps.
The above mentioned results outline certain aspects of Data Migration performance for Dynamics CRM 4.0. The combination of fast algorithms used in Microsoft Dynamics CRM 4.0 Data Migration feature, along with the new generation Intel Xeon servers results in the high throughput for complex Asynchronous Data Migration jobs.
When reading this blog, user should keep in mind the following points.
Any deployment in a similar environment may not provide the exact same numbers, but the provided numbers indicate the pattern of performance numbers for Data Migration. This can be useful in initial planning for migrating your data into CRM using Data Migration Manager.
PingBack from http://www.travel-hilarity.com/airline_travel/?p=4926
This blog post is intended to provide necessary information regarding performance of the Data Migration feature. I will provide some details on the tests and results on a specific hardware environment and some guidance on the deployment side when performin
How did you configure the mapping xml for the Complex Scenario?
For the complex scenario, I started with the Out of Box SF map. For some of the entities, like lead, opportunity, the file size was more than 32 MB. So I split them into multiple files like Lead1,Lead2. Then I modified the xml map in such a way that the mapping section of Lead is now copied two times. One is mapping for Lead1, another for Lead2 (but the mapping information is same as before). I saved the modified map and used that for migration.
You state that the machine for running DMM should not be a VM, in our environment we are running with an XP VM on 2008 with Hyper-V. Is this likely to still slow things down (given that hyper-v allows more direct hardware access)?
We did not perform migration in Hyper - V machines. But looking into some other test result executed on Hyper - V and VPC, we found that Hyper - V provides much more speed than VPC as you mentioned. Still it may not be exactly same as a dedicated physical machine. But if VPC and Hyper - V are the only alternatives, Hyper - V will be better. And if your base hardware running Hyper - V is a powerful server (HP DL series or equivalent) and you have provided sufficient memory for the guest OS (here XP), and followed other optimization tricks, migration should not be slow.
You can use SQL Server Integration Services (SSIS) to load data into the CRM. I believe this is the best way of doing that.
I made a tool that helps generates the DLL required to use from SSIS script component to write Microsoft SDK code. You can download this too for free from this location http://www.ssis4crm.com
You will fined there intsractions of doing a simple load.
This way is supported by Microsoft (using MSCRM SDK)